search

substance / texture

A visual editor for research.
MIT License
1k stars 85 forks source link

Question: How can I use my custom data-base to load archives into Texture? #529

Closed obuchtala closed 6 years ago

obuchtala commented 6 years ago

How do you dynamically load a data file instead of having it hardcoded in the source to point to the data folder? The data file would come from a database and then displayed in the Texture editor.

question asked by @samboustani

obuchtala commented 6 years ago

We have a concept of an archive which is called DAR: https://github.com/substance/dar There is a reference implementation of an HTTP implementation in node: https://github.com/substance/dar-server

The idea is that the backend provides a raw archive, which contains a manifest.xml and documents, such as JATS files, plus assets, which are allowed to be provided as URLs.

obuchtala commented 6 years ago

This is all pretty new, but should be working already. Someone that has implemented this in OJS is @kaschioudi: see https://github.com/substance/texture/issues/456 for some more details We will work on a better documentation of this in the next months.

obuchtala commented 6 years ago

Also look here: https://github.com/substance/texture/blob/master/docs/steps/0004-persistence.md#raw-archive

exportio commented 6 years ago

Thank you for your help and time, Oliver! I'll take a look and try it out. I will post a solution if it works.

exportio commented 6 years ago

I have a proof of concept on how to load an archive dynamically. However, any new archives that are created, except those that were there in the Github clone, produce the error "Uncaught (in promise) Error: File does not exist: 5af51a28070f6d1740d2ca42/manifest.xml at SimpleVFS.readFileSync (vfs.js:31)"

5af51a28070f6d1740d2ca42 being a folder under /data, same as blank and list, which both work. This new folder contains a manifest.xml and a manuscript.xml file. Is there somewhere else that it's looking?

obuchtala commented 6 years ago

@samboustani Thanks for this. I will take a look ASAP! But probably I am not able to this the next 2 days. Will ping you when I start.

exportio commented 6 years ago

I noticed at the top of vfs.js there are all the archives in a constant. Is everything hardcoded with this app? I removed the content under each line, since it was around 6 MB. I'm guessing vfs.js is being used as storage, which is why it's not finding my folder in the data folder. I'll review the documentation to see what I need to do. Thanks.

UPDATE: It looks like I have to specify the storageType inside of editor.js

const CONTENT = { "blank/manifest.xml": "blank/manuscript.xml": "elife-32671/manifest.xml": "elife-32671/manuscript.xml": "kitchen-sink/manifest.xml": "kitchen-sink/manuscript.xml": "list/manifest.xml": "list/manuscript.xml": }

obuchtala commented 6 years ago

We have the vfs (virtual fs) as a development thingie. This is generated during build from the examples, and not meant as a real solution. A real setup would be more like what we do with dar-server. Have you seen that? https://github.com/substance/texture/blob/master/make.js#L32-L40 This is again a dev setup, but there we start the dar-server with an express app (b.server is just an express instance). Maybe this helps a bit. Beyond that, maybe you can share your PoC code somewhere so that I can better understand what it looks like what you are trying :)

Cheers!

obuchtala commented 6 years ago

Another comment about vfs: we use this to simulate a real 'fs' scenario in the browser. It is actually read-only, or only in-memory read-write. I.e. everything is lost on reload. Generally almost everything can be developed and tested using the browser this way. The debugger in the browser is great, particularly in Chrome.

exportio commented 6 years ago

@oliver---- I'm passing the archive folder name as a query parameter

/Texture/index.html?archive=blank

This works fine with the original archives when cloned, but won't work on any new folders added to the data folder, because it won't find them since they're not in vfs.js file constant.

I haven't worked with dar.server yet, so I'll have to try it.

I'm wondering what the parameters are for storage, as seen below from editor.js. It is possible to pass in something other than vsf and have it handle it, or is a custom solution the only way?

let app = DevWebApp.mount({
  archiveId: substance.getQueryStringParam('archive') || 'blank',
  storageType: substance.getQueryStringParam('storage') || 'vfs',
  storageUrl: substance.getQueryStringParam('storageUrl') || '/archives'
}, window.document.body);
obuchtala commented 6 years ago

Yep, HTML param 'storage=fs' would try to read from a dar-server instance. ... or another instance compliant to that protocol. Then you have a real scenario with read-write. with storageUrl you could point it to somewhere else. But heads up: CORS

obuchtala commented 6 years ago

(sorry that needs all be documented, and isn't yet.)

obuchtala commented 6 years ago

You can get a real setup by running by 

npm start

look at package.json how it is configured, together with the block in make.js I mentioned before https://github.com/substance/texture/blob/master/package.json#L29

"node make web -s -w -d data": only the last is passed to dar-server, telling it to serve the data folder.

exportio commented 6 years ago

Unfortunately, a new error is coming up when running npm start. I even deleted the node_modules folder and ran npm install again.

Starting server on port 4000... Watching for changes... Press CTRL-C to exit. Error: listen EACCES 0.0.0.0:4000 at Object._errnoException (util.js:1022:11) at _exceptionWithHostPort (util.js:1044:20) at Server.setupListenHandle [as _listen2] (net.js:1350:19) at listenInCluster (net.js:1408:12) at Server.listen (net.js:1492:7) at Function.listen (C:\Users\Sam\Documents\GitHub\texture\node_modules\substance-bundler\dist\node_modules\express\lib\application.js:618:1) at Bundler._startServing (C:\Users\Sam\Documents\GitHub\texture\node_modules\substance-bundler\src\Bundler.js:244:17) at Bundler. (C:\Users\Sam\Documents\GitHub\texture\node_modules\substance-bundler\src\main.js:82:13) at Object.onceWrapper (events.js:313:30) at emitNone (events.js:106:13) npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! substance-texture@1.0.0-preview.3-preview.0 start: node make web -s -w -d data npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the substance-texture@1.0.0-preview.3-preview.0 start script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

2 info using npm@6.0.1 3 info using node@v8.11.1 4 verbose run-script [ 'prestart', 'start', 'poststart' ] 5 info lifecycle substance-texture@1.0.0-preview.3-preview.0~prestart: substance-texture@1.0.0-preview.3-preview.0 6 info lifecycle substance-texture@1.0.0-preview.3-preview.0~start: substance-texture@1.0.0-preview.3-preview.0 7 verbose lifecycle substance-texture@1.0.0-preview.3-preview.0~start: unsafe-perm in lifecycle true 8 verbose lifecycle substance-texture@1.0.0-preview.3-preview.0~start: PATH: C:\Users\Sam\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin;C:\Users\Sam\Documents\GitHub\texture\node_modules.bin;C:\Program Files\Docker\Docker\Resources\bin;C:\app\Sam\product\12.1.0\dbhome_1\bin;C:\app\Sam\product\11.2.0\dbhome_2\bin;C:\Python27\;C:\Python27\Scripts;C:\Program Files (x86)\Android\android-sdk-windows\platform-tools;C:\Program Files (x86)\Android\android-sdk-windows\tools;C:\Program Files (x86)\Java\jdk1.7.0_55\bin;C:\ProgramData\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Microsoft SQL Server\120\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\ManagementStudio\;C:\Program Files (x86)\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\;C:\Users\Sam\AppData\Roaming\npm;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files\Microsoft SQL Server\110\DTS\Binn\;C:\Users\Sam\AppData\Local\Programs\Python\Python35\Scripts\;C:\Users\Sam\AppData\Local\Programs\Python\Python35\;C:\Program Files (x86)\EaseUS\Todo Backup\bin;C:\Program Files (x86)\Skype\Phone\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\dotnet\;C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\PrivateAssemblies\;C:\Program Files (x86)\Microsoft SQL Server\120\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\130\DTS\Binn\;C:\Program Files\WIDCOMM\Bluetooth Software\;C:\Program Files\WIDCOMM\Bluetooth Software\syswow64;C:\Program Files\PuTTY\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\dotnet\;C:\Program Files\nodejs\;C:\Users\Sam\Anaconda2;C:\Users\Sam\Anaconda2\Scripts;C:\Users\Sam\Anaconda2\Library\bin;C:\Python27\Scripts\;C:\Users\Sam\AppData\Local\atom\bin;C:\Program Files\cURL\bin;C:\Users\Sam\AppData\Local\Microsoft\WindowsApps;C:\temp;"C:\Program Files\MySQL\MySQL Server 5.7\bin";C:\Program Files\Microsoft VS Code\bin;C:\Users\Sam.dotnet\tools;C:\Users\Sam\AppData\Local\Microsoft\WindowsApps;C:\Users\Sam\AppData\Local\GitHubDesktop\bin;C:\Program Files (x86)\EaseUS\Todo Backup\bin\x64\ 9 verbose lifecycle substance-texture@1.0.0-preview.3-preview.0~start: CWD: C:\Users\Sam\Documents\GitHub\texture 10 silly lifecycle substance-texture@1.0.0-preview.3-preview.0~start: Args: [ '/d /s /c', 'node make web -s -w -d data' ] 11 silly lifecycle substance-texture@1.0.0-preview.3-preview.0~start: Returned: code: 1 signal: null 12 info lifecycle substance-texture@1.0.0-preview.3-preview.0~start: Failed to exec start script 13 verbose stack Error: substance-texture@1.0.0-preview.3-preview.0 start: node make web -s -w -d data 13 verbose stack Exit status 1 13 verbose stack at EventEmitter. (C:\Users\Sam\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\index.js:283:16) 13 verbose stack at emitTwo (events.js:126:13) 13 verbose stack at EventEmitter.emit (events.js:214:7) 13 verbose stack at ChildProcess. (C:\Users\Sam\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\lib\spawn.js:55:14) 13 verbose stack at emitTwo (events.js:126:13) 13 verbose stack at ChildProcess.emit (events.js:214:7) 13 verbose stack at maybeClose (internal/child_process.js:925:16) 13 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5) 14 verbose pkgid substance-texture@1.0.0-preview.3-preview.0 15 verbose cwd C:\Users\Sam\Documents\GitHub\texture 16 verbose Windows_NT 10.0.17134 17 verbose argv "C:\Program Files\nodejs\node.exe" "C:\Users\Sam\AppData\Roaming\npm\node_modules\npm\bin\npm-cli.js" "start" 18 verbose node v8.11.1 19 verbose npm v6.0.1 20 error code ELIFECYCLE 21 error errno 1 22 error substance-texture@1.0.0-preview.3-preview.0 start: node make web -s -w -d data 22 error Exit status 1 23 error Failed at the substance-texture@1.0.0-preview.3-preview.0 start script. 23 error This is probably not a problem with npm. There is likely additional logging output above. 24 verbose exit [ 1, true ]

obuchtala commented 6 years ago

I guess this error means that you are not allowed to start a server on the machine you are working on. Is it maybe a some kind of shared computer, where you might need to ask a sys-admin? If it is your own computer you could try to run as root, just to identify if permissions would solve the problem.

I think it makes sense to try this with the express - Hello World example first: look here https://www.npmjs.com/package/express

obuchtala commented 6 years ago

Also make sure nothing else is running on port 4000. Maybe you have started two instances? There could be another application too. Run netstat -ano and look if something is bound to the port.

exportio commented 6 years ago

@oliver---- I started the command prompt in Admin mode after powering up this morning and it ran successfully.

As for setting the configuration in my app to use dar-server, I tried several configurations but it always returns 404 not found for the archive I point to. I'll look into it today and post my final solution to getting this to work inside another application.

obuchtala commented 6 years ago

Running as root is of course not ideal on the long run. Maybe you have restricted your user permissions too much? (I am unfortunately not a Windows permissions expert)

exportio commented 6 years ago

I got things running somewhat. The issue now is that when you run "npm start", it builds a "dist" folder. However, it does not reference the "data" folder inside the "dist" folder. It instead uses the "data" folder in the main texture root. I tested this by removing an archive folder in the root and got a 404 not found error, but the archive folder was still in the dist, which means it wasn't accessing that one.

/texture/data /texture/dist/data

What's the point of having the dist folder with the bundled application?

exportio commented 6 years ago

Sorry, I know integration is my responsibility and that the team is focusing on the editor itself. However, the NPM package page has some details that this page does not. There is a usage example (where would this code actually be executed from?) and an "examples" link that is broken. Are there actually integration examples somewhere?

https://www.npmjs.com/package/substance-texture

Usage

const { Texture } = window.texture

window.app = Texture.mount({
  documentId: 'some-doc-id',
  readXML: function(documentId, cb) {
    // fetch an XML string to be read by the editor
  },
  writeXML: function(documentId, xml, cb) {
    // write the XML string to a store
  }
}, document.body)

UPDATE: It looks like this issue was brought up before and may help by looking here: https://github.com/substance/texture/issues/115 I wish I had seen this two weeks ago.

michael commented 6 years ago

We'll work on better documentation but closing this issue to clean up a bit.