Latest stable release lacks install.sh file

[katelynsills]

Describe the bug

The production installation instructions specify using an install.sh file:

6. $ ./install.sh [destination_path] if no destination_path is provided it will default to ./prod.

However, no install.sh file is present, although one exists in the root folder of the git repo.

To Reproduce

1. Download the uwazi.tgz file that is part of the latest stable release. As of Dec-14-2022, the latest stable release is 1.97.0.
2. Unzip the file. This produces a folder called prod.
   • As a separate documentation issue, the installation documentation appears to expect this to be called uwazi.
3. Look in prod, there is no install.sh present.

Expected behavior

Production build script should also copy over install.sh from the git repo root, or installation documentation needs to be updated in some way.

Screenshots

[RafaPolit]

Thanks for the report. Indeed, there is an inconsistency. Those instructions are outdated and were meant to be run on a clone of the repo, not the production code you download on releases.

We will update accordingly.

[katelynsills]

Thank you so much! It's much appreciated!

[juandans01]

Hey Rafa, i'm facing the same issue, what would be the way of finishing the installation if install.sh is missing ? thanks in advance

[katelynsills]

Hi Rafa, I'm back from the holidays and starting to work with UWAZI again. Let me know if I can help in any way to fix this. Thanks again for your time.

[RafaPolit]

Hello. Thanks for the patience. The actual fix will take some time, so, if you need support with anything in particular, please don't hesitate to reach out, we can do a one-on-one and have this sorted quickly if something is still not working on your end.

As for this issue, we discussed this during our tech call and we have a minor update:

• Our current instructions assume you are downloading the full Source Code, but points you to the release page, where you can actually download one of three assets.
• We would prefer it if people use the already-built asset, but then, as mentioned, the install script is not only NOT included, but it would build again an asset that is already built, and other issues
• The current instructions combine install/upgrade, which is difficult to follow and also has some critical steps like yarn blank-state which, if followed during upgrade, would delete all your data. The "warning" is after the instruction to run the command, so potentially very dangerous. We believe that having separate instructions for install and upgrade is a better approach
• If we upgrade the instructions to download the already-built code, then we need to write a new .sh file that actually does the bare minimum needed to run Uwazi, which, at that point is probably only yarn blank-state and yarn migrate on install and yarn migrate on upgrade.
• The script could also reindex on request as our own migration system (??? not sure)

We will probably allocate time for this during the next week.

[RafaPolit]

Hey Rafa, i'm facing the same issue, what would be the way of finishing the installation if install.sh is missing ? thanks in advance

In theory, if you have all the required dependencies, it should be:

WARNING: Install instructions only, not upgrade! This deletes all data.

$ yarn install
$ yarn blank-state
$ yarn migrate
$ yarn reindex
$ yarn run-production

This should start Uwazi. The thing is, we need time to double check this is true with the folder structure included in the pre-built package, and that everything is correctly configured. We believe it should be, but no one has gone through the process step by step to confirm that, that is why we will address time during next week for this.

If, and ONLY IF, you don't have any data in the DB, there is no harm in trying the above instructions.

[katelynsills]

Thanks so much, Rafa!

[RafaPolit]

Work for this is being done in #5449

[RafaPolit]

The new instructions are merged into the code. Please, have a read if this is clearer and, hopefully, simpler for self hosting.

If this satisfies your current needs, please close this issue.

Thanks again for taking the time to report and follow up on the issue.

[katelynsills]

Thank you so much, @RafaPolit! I will try out the instructions today

[katelynsills]

Hi @RafaPolit, I successfully installed the dependencies, and tried out the new instructions, but I'm getting an error when running the yarn blank-state step. Also, I'm not sure if it's pertinent, but npm config set scripts-prepend-node-path auto in the dependency instructions also failed. Please let me know if I can provide any more info or can be helpful in any way.

npm config set scripts-prepend-node-path auto logs:

katesills@kates-mbp ~ % npm config set scripts-prepend-node-path auto
npm ERR! `scripts-prepend-node-path` is not a valid npm option

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/katesills/.npm/_logs/2023-01-25T23_44_56_802Z-debug-0.log
katesills@kates-mbp ~ % cat /Users/katesills/.npm/_logs/2023-01-25T23_44_56_802Z-debug-0.log
0 verbose cli /Users/katesills/.nvm/versions/node/v18.13.0/bin/node /Users/katesills/.nvm/versions/node/v18.13.0/bin/npm
1 info using npm@9.4.0
2 info using node@v18.13.0
3 timing npm:load:whichnode Completed in 1ms
4 timing config:load:defaults Completed in 1ms
5 timing config:load:file:/Users/katesills/.nvm/versions/node/v18.13.0/lib/node_modules/npm/npmrc Completed in 0ms
6 timing config:load:builtin Completed in 1ms
7 timing config:load:cli Completed in 0ms
8 timing config:load:env Completed in 1ms
9 timing config:load:project Completed in 0ms
10 timing config:load:file:/Users/katesills/.npmrc Completed in 1ms
11 timing config:load:user Completed in 2ms
12 timing config:load:file:/Users/katesills/.nvm/versions/node/v18.13.0/etc/npmrc Completed in 0ms
13 timing config:load:global Completed in 0ms
14 timing config:load:setEnvs Completed in 1ms
15 timing config:load Completed in 6ms
16 timing npm:load:configload Completed in 6ms
17 timing npm:load:mkdirpcache Completed in 0ms
18 timing npm:load:mkdirplogs Completed in 0ms
19 verbose title npm config set scripts-prepend-node-path auto
20 verbose argv "config" "set" "scripts-prepend-node-path" "auto"
21 timing npm:load:setTitle Completed in 16ms
22 timing config:load:flatten Completed in 2ms
23 timing npm:load:display Completed in 2ms
24 verbose logfile logs-max:10 dir:/Users/katesills/.npm/_logs/2023-01-25T23_44_56_802Z-
25 verbose logfile /Users/katesills/.npm/_logs/2023-01-25T23_44_56_802Z-debug-0.log
26 timing npm:load:logFile Completed in 4ms
27 timing npm:load:timers Completed in 0ms
28 timing npm:load:configScope Completed in 0ms
29 timing npm:load Completed in 29ms
30 info config set "scripts-prepend-node-path" "auto"
31 timing command:config Completed in 2ms
32 verbose stack Error: `scripts-prepend-node-path` is not a valid npm option
32 verbose stack     at Config.set (/Users/katesills/.nvm/versions/node/v18.13.0/lib/node_modules/npm/lib/commands/config.js:158:15)
32 verbose stack     at Config.exec (/Users/katesills/.nvm/versions/node/v18.13.0/lib/node_modules/npm/lib/commands/config.js:120:22)
32 verbose stack     at Config.cmdExec (/Users/katesills/.nvm/versions/node/v18.13.0/lib/node_modules/npm/lib/base-command.js:130:17)
32 verbose stack     at Npm.exec (/Users/katesills/.nvm/versions/node/v18.13.0/lib/node_modules/npm/lib/npm.js:154:20)
32 verbose stack     at async module.exports (/Users/katesills/.nvm/versions/node/v18.13.0/lib/node_modules/npm/lib/cli.js:134:5)
33 verbose cwd /Users/katesills
34 verbose Darwin 22.2.0
35 verbose node v18.13.0
36 verbose npm  v9.4.0
37 error `scripts-prepend-node-path` is not a valid npm option
38 verbose exit 1
39 timing npm Completed in 40ms
40 verbose code 1
41 error A complete log of this run can be found in:
41 error     /Users/katesills/.npm/_logs/2023-01-25T23_44_56_802Z-debug-0.log

yarn blank-state logs:

katesills@kates-mbp uwazi % cd latest
katesills@kates-mbp latest % NODE_ENV=production yarn blank-state productiondb
yarn run v1.22.19
warning ../../../../../package.json: No license field
$ ./database/blank_state.sh productiondb

Deleting productiondb database on 127.0.0.1
MongoDB shell version v4.2.23
connecting to: mongodb://127.0.0.1:27017/productiondb?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("5a89ee73-f347-42c1-ae51-55aede391aa5") }
MongoDB server version: 4.2.23
{ "ok" : 1 }
2023-01-25T15:57:20.164-0800    the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead
2023-01-25T15:57:20.164-0800    building a list of collections to restore from blank_state/uwazi_development dir
2023-01-25T15:57:20.166-0800    reading metadata for productiondb.translations from blank_state/uwazi_development/translations.metadata.json
2023-01-25T15:57:20.166-0800    reading metadata for productiondb.updatelogs from blank_state/uwazi_development/updatelogs.metadata.json
2023-01-25T15:57:20.166-0800    reading metadata for productiondb.migrations from blank_state/uwazi_development/migrations.metadata.json
2023-01-25T15:57:20.167-0800    reading metadata for productiondb.templates from blank_state/uwazi_development/templates.metadata.json
2023-01-25T15:57:20.253-0800    restoring productiondb.updatelogs from blank_state/uwazi_development/updatelogs.bson
2023-01-25T15:57:20.334-0800    restoring productiondb.translations from blank_state/uwazi_development/translations.bson
2023-01-25T15:57:20.335-0800    restoring indexes for collection productiondb.updatelogs from metadata
2023-01-25T15:57:20.337-0800    no indexes to restore
2023-01-25T15:57:20.337-0800    finished restoring productiondb.translations (1 document, 0 failures)
2023-01-25T15:57:20.337-0800    reading metadata for productiondb.settings from blank_state/uwazi_development/settings.metadata.json
2023-01-25T15:57:20.557-0800    restoring productiondb.migrations from blank_state/uwazi_development/migrations.bson
2023-01-25T15:57:20.561-0800    no indexes to restore
2023-01-25T15:57:20.561-0800    finished restoring productiondb.migrations (110 documents, 0 failures)
2023-01-25T15:57:20.561-0800    reading metadata for productiondb.users from blank_state/uwazi_development/users.metadata.json
2023-01-25T15:57:20.598-0800    restoring productiondb.templates from blank_state/uwazi_development/templates.bson
2023-01-25T15:57:20.608-0800    no indexes to restore
2023-01-25T15:57:20.608-0800    finished restoring productiondb.templates (1 document, 0 failures)
2023-01-25T15:57:20.608-0800    reading metadata for productiondb.relationtypes from blank_state/uwazi_development/relationtypes.metadata.json
2023-01-25T15:57:20.681-0800    restoring productiondb.settings from blank_state/uwazi_development/settings.bson
2023-01-25T15:57:20.692-0800    no indexes to restore
2023-01-25T15:57:20.692-0800    finished restoring productiondb.settings (1 document, 0 failures)
2023-01-25T15:57:20.692-0800    reading metadata for productiondb.sessions from blank_state/uwazi_development/sessions.metadata.json
2023-01-25T15:57:20.918-0800    restoring productiondb.users from blank_state/uwazi_development/users.bson
2023-01-25T15:57:20.920-0800    restoring indexes for collection productiondb.users from metadata
2023-01-25T15:57:20.957-0800    restoring productiondb.relationtypes from blank_state/uwazi_development/relationtypes.bson
2023-01-25T15:57:20.958-0800    no indexes to restore
2023-01-25T15:57:20.958-0800    finished restoring productiondb.relationtypes (0 documents, 0 failures)
2023-01-25T15:57:20.958-0800    reading metadata for productiondb.files from blank_state/uwazi_development/files.metadata.json
2023-01-25T15:57:21.040-0800    restoring productiondb.sessions from blank_state/uwazi_development/sessions.bson
2023-01-25T15:57:21.041-0800    restoring indexes for collection productiondb.sessions from metadata
2023-01-25T15:57:21.259-0800    restoring productiondb.files from blank_state/uwazi_development/files.bson
2023-01-25T15:57:21.259-0800    restoring indexes for collection productiondb.files from metadata
2023-01-25T15:57:21.700-0800    finished restoring productiondb.sessions (0 documents, 0 failures)
2023-01-25T15:57:21.700-0800    reading metadata for productiondb.entities from blank_state/uwazi_development/entities.metadata.json
2023-01-25T15:57:21.719-0800    finished restoring productiondb.updatelogs (96 documents, 0 failures)
2023-01-25T15:57:21.719-0800    reading metadata for productiondb.dictionaries from blank_state/uwazi_development/dictionaries.metadata.json
2023-01-25T15:57:21.962-0800    restoring productiondb.entities from blank_state/uwazi_development/entities.bson
2023-01-25T15:57:21.963-0800    restoring indexes for collection productiondb.entities from metadata
2023-01-25T15:57:22.003-0800    restoring productiondb.dictionaries from blank_state/uwazi_development/dictionaries.bson
2023-01-25T15:57:22.011-0800    no indexes to restore
2023-01-25T15:57:22.011-0800    finished restoring productiondb.dictionaries (0 documents, 0 failures)
2023-01-25T15:57:22.011-0800    reading metadata for productiondb.connections from blank_state/uwazi_development/connections.metadata.json
2023-01-25T15:57:22.304-0800    restoring productiondb.connections from blank_state/uwazi_development/connections.bson
2023-01-25T15:57:22.313-0800    restoring indexes for collection productiondb.connections from metadata
2023-01-25T15:57:22.541-0800    finished restoring productiondb.files (0 documents, 0 failures)
2023-01-25T15:57:22.541-0800    reading metadata for productiondb.activitylogs from blank_state/uwazi_development/activitylogs.metadata.json
2023-01-25T15:57:22.778-0800    restoring productiondb.activitylogs from blank_state/uwazi_development/activitylogs.bson
2023-01-25T15:57:22.779-0800    restoring indexes for collection productiondb.activitylogs from metadata
2023-01-25T15:57:23.327-0800    finished restoring productiondb.users (1 document, 0 failures)
2023-01-25T15:57:23.386-0800    finished restoring productiondb.activitylogs (0 documents, 0 failures)
2023-01-25T15:57:23.784-0800    finished restoring productiondb.entities (0 documents, 0 failures)
2023-01-25T15:57:23.989-0800    finished restoring productiondb.connections (0 documents, 0 failures)
2023-01-25T15:57:23.989-0800    210 document(s) restored successfully. 0 document(s) failed to restore.
warning ../../../../../package.json: No license field
$ node scripts/run.js ../scripts/migrate.js
add_system_key_translations...
update_translations_of_settings_tooltips...
add_system_key_translations...
add_system_key_translations...
add_system_key_translations...
add_system_key_translations...
update_problematic_UI_translations...
add_system_key_translations...
reset-files-stuck-in-processing...
warning ../../../../../package.json: No license field
$ node scripts/run.js ../database/reindex_elastic.js
Deleting index productiondb...
The index was not found:
{
 "name": "FetchResponseError",
 "json": {
  "error": {
   "root_cause": [
    {
     "type": "index_not_found_exception",
     "reason": "no such index [productiondb]",
     "resource.type": "index_or_alias",
     "resource.id": "productiondb",
     "index_uuid": "_na_",
     "index": "productiondb"
    }
   ],
   "type": "index_not_found_exception",
   "reason": "no such index [productiondb]",
   "resource.type": "index_or_alias",
   "resource.id": "productiondb",
   "index_uuid": "_na_",
   "index": "productiondb"
  },
  "status": 404
 },
 "status": 404,
 "headers": {}
}
Moving on.
 [done]
Creating index productiondb...
 - Base properties mapping
2023-01-25T23:57:57.357Z [productiondb] Uncaught Reindex error.
Fetch returned a response with status 503.
{
 "name": "FetchResponseError",
 "json": {
  "error": {
   "root_cause": [
    {
     "type": "uncategorized_execution_exception",
     "reason": "Failed execution"
    }
   ],
   "type": "master_not_discovered_exception",
   "reason": "FailedToCommitClusterStateException[publication failed]; nested: UncategorizedExecutionException[Failed execution]; nested: ExecutionException[java.nio.file.NoSuchFileException: /Users/katesills/code/consulting-2022/starling/elasticsearch-7.17.6/data/nodes/0/_state/write.lock]; nested: NoSuchFileException[/Users/katesills/code/consulting-2022/starling/elasticsearch-7.17.6/data/nodes/0/_state/write.lock];",
   "caused_by": {
    "type": "failed_to_commit_cluster_state_exception",
    "reason": "publication failed",
    "caused_by": {
     "type": "uncategorized_execution_exception",
     "reason": "Failed execution",
     "caused_by": {
      "type": "execution_exception",
      "reason": "java.nio.file.NoSuchFileException: /Users/katesills/code/consulting-2022/starling/elasticsearch-7.17.6/data/nodes/0/_state/write.lock",
      "caused_by": {
       "type": "no_such_file_exception",
       "reason": "/Users/katesills/code/consulting-2022/starling/elasticsearch-7.17.6/data/nodes/0/_state/write.lock"
      }
     }
    }
   }
  },
  "status": 503
 },
 "status": 503,
 "headers": {}
}
Will exit with (1)

Restoring index settings... [done]
2023-01-25T23:58:27.386Z [productiondb] uncaughtException: Fetch returned a response with status 503.
FetchResponseError: Fetch returned a response with status 503.
    at /Users/katesills/code/consulting-2022/starling/uwazi-2023-installation/uwazi/latest/app/shared/JSONRequest.js:128:13
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /Users/katesills/code/consulting-2022/starling/uwazi-2023-installation/uwazi/latest/database/reindex_elastic.js:42:7
[Tenant error] Error: Accessing nonexistent async context
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.

[katelynsills]

Should there be a yarn install within latest as well before running node server.js?

[RafaPolit]

Apparently there shouldn't be a need for that. Checking this out.

[RafaPolit]

Let's break this up into different areas:

• The scripts-prepend-node-path may be a legacy option, I find documentation of it until NPM 6.14.18, but not on newer builds, so let's skip that for the time being.
• The index_not_found_exception is expected. We should look into hiding that warning, but the first time setting up Uwazi it tries to delete the index and then create it. The deletion fails due to the index not existing, but then it is correctly created, so that error is expected.
• The master_not_discovered_exception is the real key problem here. This suggests a problem with the Elastic Search configuration. Apparently the Elastic nodes are not able to ping each other? This is a bit beyond our ability to help, but some ES debugging like sharing the output of curl localhost:9200 could be useful? The communication between ES nodes happens on port 9300 by default, maybe try seeing if the nodes can reach each other? How is ES configured? As a sharded cluster? A simple node? Does the ES service reside on the same machine as the Node and Mongo services?

Basically, all appears to be working as expected from the Uwazi end, there seems to be some problem with ES and/or the way the services communicate to it.

[katelynsills]

Thanks so much! I tried installing again, and this time it worked! Yesterday I had checked my basic ES setup with curl -X GET "localhost:9200/?pretty" and that returned the usual, so I'm at a loss for what it was, but maybe ES got shut down somehow.

I think we can close this! Thank you again!

[RafaPolit]

Glad that things are working as expected. Please don't hesitate to share any further questions. Thanks for all the contributions.