Trouble installing on a Linux server

[geyerbri]

Hello, I'm trying to setup an install of Voyager for a tech lab at Michigan State University (@msu-dhi-lab). I cannot figure out what I am doing wrong during the install process.

My server is a DigitalOcean droplet with Ubuntu 22.04. I have also installed NodeJS, NPM, and Apache2 (for eventual access via a browser). Just to confirm, I did successfully load the Apache2 index.html file via my browser. Additionally, I have tried to install both with and without Apache2 installed on the server.

I've failed while both creating the voyager directory both at server root, and at /var/www/html (when trying with Apache2 installed). In each instance, I've copied the droplet's IP address into the .env file on the line "VOYAGER_SERVER_IPV4_ADDR" (I don't have a domain set up for it yet) and leave the port as the default.

I tried following the non-Docker installation instructions in all cases:

npm install (this finishes, but with a bunch of deprecation warnings and 8 moderate severity vulnerabilities)

then

npm run build-server

followed finally by

npm run build-prod

After running the last command, it appears to be successfully running in console, but when using my browser to visit http://[IP-ADDRESS]:8000, nothing loads and I get the error "ERR_CONNECTION_RESET". I don't believe there's a firewall blocking the port.

I have also tried visiting http://[IP-ADDRESS]:8000/voyager-explorer.html and http://[IP-ADDRESS]:8000/dist/voyager-explorer.html, but none load (I did this because I can't tell what web address I would visit in my browser to view one of the applications working).

Am I missing something during the setup or installation process? Using Docker is not an option for me, but I did try a test on my own personal droplet and had the same problem with nothing loading in my browser.

[gjcope]

Hi @geyerbri , I don't have any specific experience with that environment (one of the reasons we recommend Docker) but a few general questions:

• Are you planning to do development work with Voyager? If not, I would recommend taking a look at https://smithsonian.github.io/dpo-voyager/introduction/hosting/ - Since Voyager is just a web component, it doesn't require any real installation to use, just a web server and access to a WebDAV enabled location for the source files.
• The commands listed above build the included express server and web component, but are you also starting the server? Try "npm run start" or "npm run server".
• The Voyager distribution (built or pre-built) includes a set of example html pages showing how you can embed the various components. You can use these or embed the component in your own page to see it working. If you were using the included express server to serve the pages, the url would be something like http://[IP-ADDRESS]:8000/voyager-explorer.html

[geyerbri]

Thanks for getting back to me. Let me apologize in advance for all of the following detail. I don't know what's relevant so I'm trying to be thorough in trying things and then reporting the outcome.

I think I have a slightly better handle on what my issue is. It turns out, I haven't been watching the terminal long enough after running the final npm command I reported - npm run build-prod - so I wasn't catching that in my case this command wasn't actually completing. Because of my lack of knowledge, I was assuming that this command was starting the server and so it wouldn't complete and return to the terminal prompt.

After waiting a longer time, it finally showed "killed" and then returned to the prompt. Here's what the console showed after running the build-prod command:

root@voyager:/var/www/voyager# npm run build-prod

> voyager@0.37.0 build-prod
> cd source/client; webpack --mode=production --env=app=all; cd ../..

VOYAGER - WEBPACK BUILD SCRIPT
  application = voyager-explorer
  mode = production
  offline = false
  version = v0.39.0
  source directory = /var/www/voyager/source
  output directory = /var/www/voyager/dist

VOYAGER - WEBPACK BUILD SCRIPT
  application = voyager-mini
  mode = production
  offline = false
  version = v0.39.0
  source directory = /var/www/voyager/source
  output directory = /var/www/voyager/dist

VOYAGER - WEBPACK BUILD SCRIPT
  application = voyager-launcher
  mode = production
  offline = false
  version = v0.39.0
  source directory = /var/www/voyager/source
  output directory = /var/www/voyager/dist

VOYAGER - WEBPACK BUILD SCRIPT
  application = voyager-story
  mode = production
  offline = false
  version = v0.39.0
  source directory = /var/www/voyager/source
  output directory = /var/www/voyager/dist
Browserslist: caniuse-lite is outdated. Please run:
  npx browserslist@latest --update-db
  Why you should do it regularly: https://github.com/browserslist/browserslist#browsers-data-updating
Killed
root@voyager:/var/www/voyager#

(I cloned the current version of the repo, so I don't know why it starts by saying it's v0.37.0 when the current repo is v0.39.0.)

Per the part right before the "Killed" line, I ran npx browserslist@latest --update-db, then tried running npm run build-prod again, but it hangs after again showing the four "VOYAGER - WEBPACK BUILD SCRIPT" blocks,, then showing "Killed" and ending itself. After checking the "dist" directory, I can see that the only html file it created is "voyager-mini.html", so it appears to not be finishing the process. Even so, I still tried npm run start to start up the server (which I hadn't done before you pointed this out). The server starts up ok. When I visit voyager-mini.html via browser, it 'resolves' (so no 404 error or anything) but it's blank and the console shows this one error:

Uncaught Error: document schema validation failed
    at qv.openDocument (voyager-mini.min.js:52:138002)
    at Xv.createDocument (voyager-mini.min.js:52:141387)
    at new My (voyager-mini.min.js:55:13938)
    at Ay.firstConnected (voyager-mini.min.js:55:16737)
    at Ay.connectedCallback (voyager-mini.min.js:8:75046)
    at voyager-mini.min.js:8:75474
    at Ty (voyager-mini.min.js:55:16201)
    at voyager-mini.min.js:55:16814
    at voyager-mini.min.js:55:16855
    at voyager-mini.min.js:55:16859

Is the build command not working properly, possibly due to an issue with my server droplet (or with nodejs) not having enough allocated resources? The droplet is configured with 2GB of RAM and one hyper-thread allocated for its CPU (this is how DigitalOcean handles their basic virtual servers). What are the system requirements for the nodejs app, and/or for the express server?

Also just fyi, I haven't ignored your suggestion to just host the pre-built web component, but I'm also unsure how WebDAV would work in my case (because I don't know about it), so to simplify things I wanted to rely on the express server to try out Story. The self-hosting instructions page for Story says that to rely on the express server I'll need to do this install process. Following your suggestion though, I did spin up a server with just the web component (served with Apache2) and finally got the empty voyager-explorer.html file to load, as well as voyager-story.html?mode=standalone. I could upload gbl files to pose them (obj versions of the same models wouldn't upload though, are objs supported?) and save the project as a zip containing the models and an svx.json file.

[gjcope]

I've never seen the 'Killed' message before, but doing a little research it looks like you are right and that this is relatively common with DigitalOcean servers when Webpack runs out of memory. A couple of ideas:

• You can try building Explorer or Story independently with a command like: cd source/client; webpack --mode=production --env=app=explorer; cd ../.. - just swap app type for whatever you want to build.

• You can try allotting more memory to Node with the 'max-old-space-size' flag like cd source/client; node --max-old-space-size=8192 ../../node_modules/webpack/bin/webpack.js --mode=production --env=app=explorer; cd ../.. but my understanding is that the default is 2GB already, so it's not clear to me that this will help in your situation.

Re: .obj files, Voyager does support them but the standalone version doesn't currently support dragging and dropping that file type. You'd have to manually add it to the scene file. We are trying to encourage use of web friendly formats.

[geyerbri]

Ok it sounds like Voyager's system requirements (particularly RAM), whatever they might be, are larger than what are currently allocated to my virtual server/droplet. If you're able to share these requirements, I'd still like to know them so that we can make a cost/benefit decision about having a setup that meets the minimum requirements for the express server.

Given the system req limitations (and not knowing how much RAM I'd need to add to my current droplet to get it to work properly), I'm instead teaching myself about WebDAV. I have it appropriately configured for a directory on the same server as a Voyager pre-build and have set up authentication; the WebDAV enabled directory is a subdirectory of dist, alongside all the voyager html files. How do I make the connection between that WebDAV directory and Story? Is there some bit of html code to be added to voyager-story.html? I've already tried simply including the directory as the defined root variable, both as a URL variable (i.e. https://[IP-ADDRESS]/dist/voyager-story-dev.html?root=assets) and as an attribute defined in the page's html (i.e. <voyager-story root="assets"></voyager-story>). I have also tried making the dist directory itself the WebDAV enabled directory.

[gjcope]

Were you able to try building an individual app with the commands I listed above? This will require significantly less memory and should definitely work with less than 2GB. It's hard to say exactly what the requirements would be as some of that is platform dependent but I can try to come up with something. On my platform, on out of memory error will be thrown that provides some insight.

For the WebDAV piece there's no code to be added, it's just setting the root variable appropriately. What happens when you do that? Are you pointing at a location that has a Voyager scene document? In your example, you would want to put a '/' at the end to indicate that it is a directory, but other than that it looks good. Have you checked for errors in the browser console?

[geyerbri]

To get Voyager to install without significant errors, I had to make sure nodejs was v14, but DigitalOcean provides v12 through apt. But by using an alternative source for v14 (and the included npm version, see https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-ubuntu-22-04 if you want to understand what I'm talking about in more detail), somehow I'm not able to then install webpack to use it for building apps individually. I get errors saying that dependencies can't be installed. So I'm stuck on that front.

For WebDAV and Story, I don't know what you mean by pointing at a scene document. Isn't Story for creating the scene document in the first place? ("The Voyager Story tool is used to author Voyager scenes and documents.") I think I clearly don't understand how the tool works at all, which is why I don't understand why I'd need to supply it with an initial scene document, given that I DON'T supply it a scene document when I open it in Standalone mode. Is there documentation somewhere that specifically notes which of the available properties (root, document, model, geometry, texture, quality, referrer, mode) are required for Story to start up correctly in an express server or WebDAV environment and not in Standalone mode?

[gjcope]

We are woefully behind on Node.js version support so I can't offer any help with potential v14 issues, but hopefully soon.

My question was meant to get a better understanding of what you were trying to do. Without knowing the specifics and any errors you may be getting in the browser, it's tough to speculate on what the issue is.

As originally intended, no, Story was not meant to create the core scene document that identifies the models you want to load. It was meant to author the experience. Pose the model, add educational content, etc., as documented here: https://smithsonian.github.io/dpo-voyager/story/usage/ . Our original pipeline relies on our Cook workflow orchestration tool which generates the scene file.

As the tool has evolved and gained more users, it became clear that we needed a way for these scene files to be created outside of Cook. Currently this comes in the form of Voyager 'Standalone' mode or adding the dragdrop attribute to the standard mode that will allow you to drag and drop models. There's documentation here (https://smithsonian.github.io/dpo-voyager/explorer/scene-create/) on how to get/create a scene file, except for the dragdrop method because we don't support it internally due to the security implications.

There really aren't any 'required' attributes, it all depends on your goal. If you were trying to edit an existing scene for example, it would be necessary to provide the 'root' and 'document' attributes so that the scene can be found. There are examples of this in the docs.

We are a small team and always working to improve. If you have any feedback on better new user documentation please let us know, or better yet feel free to make a PR directly and we will include it!

[gjcope]

Following up again with a little memory usage investigation:

With the setup used for the Docker image, building has the following top usage:

• build-dev: ~2 GB
• build-prod: ~2.8 GB

There is an optimization forthcoming from @sdumetz (#242) that drops to:

• build-dev ~700MB
• build-prod: ~2.8 GB

The memory usage when building for production is mainly coming from the terser plugin that does minification of the code. It seems to be a well documented issue. If you change the plugin's 'parallel' parameter to false (stopping the generation of worker threads) it drops build-prod to ~1.8 GB but also increases build time by about 50% (in my test case).

[geyerbri]

Hi @jcope, thank you very much for putting a ton of work into this. My apologies if the tone of my last comment was frustrated, as that was entirely directed at my own lack of knowledge and skills but I suspect it came across as more broad than that.

With the info about intentions, workflow, RAM reqs, and the possibility of dropping the required RAM by disabling parallel for terser, I've been able to piece together a rough idea of how parts work together and evaluate it for our needs.

So to give you more info about our intended use (I should have done this earlier, so please accept another apology from me), we were looking at the possibility of using Packrat to standup and manage a 3D collections repository, with a variety of filetypes, possibly including obj and stl. Then, we were wondering if it would be possible to standup an instance of Voyager Story, then have the 3d models (glb, gltf, obj, and if possible stl) from Packrat's directories available to select and import into a blank scene for posing, lighting, etc., and then that scene with models could be exported with a unique filename to a folder on the server, so it could be embedded using Explorer on a website. Due to our resource limitations here, we are unable to use Cook for initially creating scenes instead.

While I know Story wasn't designed to do this at all and I knew this from the beginning, it is now obvious to me that it isn't really possible to 'hack' it together either and have the tool work via the web interface in the way we wanted. But I couldn't figure that out until I had some idea how Story worked when not running in Standalone mode, which is why I've been trying so hard to get it up and running.

Following the revelation about the RAM reqs (even disabling the parallel parameter didn't drop the req low enough, because the DigitalOcean droplet only has about 1.5GB free), I temporarily upped the RAM to meet the reqs to get the build to complete. I was finally able to familiarize myself with Story enough to know I won't be able to get it working how we were thinking.

One final thing, about NodeJS, is that I was able to get the installation to complete when my version was either 14 or 16, but none of the others, as an fyi. With my limited knowledge I am still unable to get webpack installed when my NodeJS version is different from DigitalOcean's default though, so that is why I ultimately could not follow your advice to build the tools individually. I don't know enough to understand why the build worked when using npm and your authored processes, but not webpack on its own, but the details probably don't matter.

[gjcope]

Hi @geyerbri , no worries at all.

I'm glad to hear you finally got it to build and appreciate the extra info. Hopefully we'll have time to streamline that build process in the near future, but it will probably still be geared towards Docker for broader use.

What you propose actually sounds doable to me. We are doing something similar with Packrat right now, albeit not with empty scene files but that shouldn't make much of a difference. To put together something custom, the key function to take a look at would probably be https://github.com/Smithsonian/dpo-voyager/blob/e4efdba89a6ea124cd1b99b713f7571722cc8b38/source/client/components/CVMediaManager.ts#L116

Or there are some hackier ways it could happen if you don't want to get that deep (like using the 'model' url parameter). If it's something you are still interested in pursuing, I'd be happy have a conversation offline. https://dpo.si.edu/contact

[umer936]

For anyone who also struggles on it showing an empty page, it's user error from not RTFM.

Make sure you have the URL set with a view and parameters matching a model in your files/ directory: http://localhost:8000/voyager-story-dev.html?root=models/tusk/&document=tusk.svx.json

After that it'll work.