Documentation

[opsecx]

Hi, I'm missing some documentation on how to configure this towards a self-hosted postgreSQL. I assume env.example is put into .env. The post and pre rpc urls I assume is relating to the Shielded Expedition post and pre-fork, how do these correspond to current new environments (0.41 which is not the SE testnet). Thanks

[egasimus]

PRE_UNDEXER_RPC_URL and POST_UNDEXER_RPC_URL can be used to handle any fork or chain split that occurs at a fixed block height, as determined by NODE_LOWEST_BLOCK_HEIGHT.

If there is no chain split, but the new version restarts from block 1 (as has been the case with the most recent few versions), you can just set PRE/POST to the same value and ignore NODE_LOWEST_BLOCK_HEIGHT.

We haven't tested with 0.41 yet, but you can try it anyway and tell us what works and what doesn't - it'll help us update faster :slightly_smiling_face:

[opsecx]

PRE_UNDEXER_RPC_URL and POST_UNDEXER_RPC_URL can be used to handle any fork or chain split that occurs at a fixed block height, as determined by NODE_LOWEST_BLOCK_HEIGHT.

If there is no chain split, but the new version restarts from block 1 (as has been the case with the most recent few versions), you can just set PRE/POST to the same value and ignore NODE_LOWEST_BLOCK_HEIGHT.

We haven't tested with 0.41 yet, but you can try it anyway and tell us what works and what doesn't - it'll help us update faster :slightly_smiling_face:

Thanks. Will it be best to pull directly from main source branch (stable?) or go with latest release?

[egasimus]

You can go with main; I've just updated it with the latest commits. v2.0.0 is still on Namada 0.39 I think

[opsecx]

You can go with main; I've just updated it with the latest commits. v2.0.0 is still on Namada 0.39 I think

Thanks.

`cd fadroma/packages/namada && npm run build:wasm:dev

> @fadroma/namada@1.0.0-rc.27 build:wasm:dev > wasm-pack build --dev --target web && rm -v pkg/package.json pkg/.gitignore

sh: 1: wasm-pack: not found`

Does documentation need updating, as this does not seem to be working? (I updated submodules)

[opsecx]

Would also be great with some docs on exactly which command to run after to start indexing, that's not entirely clear from docs atm

[egasimus]

You're absolutely correct that the documentation could use improvement. We'll add these things to the README.

You can go with main; I've just updated it with the latest commits. v2.0.0 is still on Namada 0.39 I think

Thanks.

`cd fadroma/packages/namada && npm run build:wasm:dev

@fadroma/namada@1.0.0-rc.27 build:wasm:dev > wasm-pack build --dev --target web && rm -v pkg/package.json pkg/.gitignore

sh: 1: wasm-pack: not found`

Does documentation need updating, as this does not seem to be working? (I updated submodules)

You need the Rust WASM toolchain (cargo install wasm-pack)

Would also be great with some docs on exactly which command to run after to start indexing, that's not entirely clear from docs atm

just up will run indexer, API server, and database, through docker compose. If you don't want to go through docker you can run ./undexer index and ./undexer api.

Heads up - the OpenAPI spec might also be out of date, check src/routes.js for actual routes. (I'm considering migrating the API server to some REST framework that generates the API spec automatically, but right now we have what we have :slightly_smiling_face: )

[opsecx]

thanks. looks from wasm compilation like tendermint 0.34.1 and namada 0.40 are being used?

[opsecx]

Tbh I keep getting error after error here. About to run away screaming in frustration 😂 (jk but in seriousness some thought needs to be given to the useability-side of this...)

[opsecx]

./undexer index node:internal/modules/esm/resolve:854 throw new ERR_MODULE_NOT_FOUND(packageName, fileURLToPath(base), null); (and then a lot of debug lines)

[egasimus]

Oh my oh my.

You should've pasted the debug lines (stack trace) too, because those are the ones that let us know what the problem is :slightly_smiling_face: You still can do that, if you want, and it will be appreciated.

Have you tried the Docker Compose workflow? After building the WASM, run docker compose up and it should all bootstrap itself into place.

Anyway, I totally agree that the user experience can, and should, be improved. So far we've only had opportunity to optimize for developer experience - somewhat :sweat_smile: But now, thanks to your feedback, I know we could add e.g. a container that builds the WASM, so people don't have to do that manually.

[opsecx]

I would strongly urge to have a non-docker setup available. I much prefer that personally. The difficulty so far has not been compiling etc but lack of any instructions.. (compiling went fint once I wasn't searching in the dark but knew what to do:)) I'll post the debug in a little bit

[opsecx]

./undexer index node:internal/modules/esm/resolve:854 throw new ERR_MODULE_NOT_FOUND(packageName, fileURLToPath(base), null); ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@hackbg/logs' imported from /home/(user)/(path)/undexer at packageResolve (node:internal/modules/esm/resolve:854:9) at moduleResolve (node:internal/modules/esm/resolve:927:18) at defaultResolve (node:internal/modules/esm/resolve:1157:11) at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:383:12) at ModuleLoader.resolve (node:internal/modules/esm/loader:352:25) at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:227:38) at ModuleWrap. (node:internal/modules/esm/module_job:87:39) at link (node:internal/modules/esm/module_job:86:36) { code: 'ERR_MODULE_NOT_FOUND' }

Node.js v20.15.0

[egasimus]

Without Docker it's probably more steps and less portable :shrug: But I do see your point - gonna do a fresh checkout of the code and document how to get it up an running without containers

[egasimus]

And in your case, I think you need to run pnpm i :slightly_smiling_face:

[opsecx]

Without Docker it's probably more steps and less portable :shrug: But I do see your point - gonna do a fresh checkout of the code and document how to get it up an running without containers

Good stuff, let me know if you update anything in the codebase so I need to repull.

[opsecx]

And in your case, I think you need to run pnpm i :slightly_smiling_face:

you mean npm i? (which I already tried and which throws yet another error..)

[opsecx]

alright got it, installing now..

[opsecx]

getting this warn on pnpm i: WARN  Failed to create bin at /home/(user)/undexer/fadroma/node_modules/.bin/docs. ENOENT: no such file or directory, open '/home/(user)/undexer/fadroma/node_modules/@hackbg/docs/docs.cli.cjs'

putting it here because I get yet another error in next step when running indexer (next post)

[opsecx]

./undexer index ⏳ Starting @hackbg/undexer 2.0.0... ⏳ Compiling TypeScript... ⌛ Compiled TypeScript in 0.087s ! DB FIXME: CREATE COLLATION threw. ✔ Undexer ⏳ Starting at 2024-07-29T10:30:52.266Z ✔ Undexer ⏳ Patching globalThis.fetch... ✔ Undexer ⏳ Syncing DB schema... /home/(user)/undexer/node_modules/.pnpm/sequelize@6.37.3_pg@8.12.0/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:149 reject(new sequelizeErrors.ConnectionError(err)); ^

ConnectionError [SequelizeConnectionError]: SASL: SCRAM-SERVER-FIRST-MESSAGE: client password must be a string at Client._connectionCallback (/home/(user)/undexer/node_modules/.pnpm/sequelize@6.37.3_pg@8.12.0/node_modules/sequelize/lib/dialects/postgres/connection-manager.js:149:20) at Client._handleErrorWhileConnecting (/home/(user)/undexer/node_modules/.pnpm/pg@8.12.0/node_modules/pg/lib/client.js:327:19) at Client._handleErrorEvent (/home/(user)/undexer/node_modules/.pnpm/pg@8.12.0/node_modules/pg/lib/client.js:337:19) at Connection.emit (node:events:519:28) at Client._handleAuthSASLContinue (/home/(user)/undexer/node_modules/.pnpm/pg@8.12.0/node_modules/pg/lib/client.js:275:23) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) { parent: Error: SASL: SCRAM-SERVER-FIRST-MESSAGE: client password must be a string at Object.continueSession (/home/(user)/undexer/node_modules/.pnpm/pg@8.12.0/node_modules/pg/lib/crypto/sasl.js:24:11) at Client._handleAuthSASLContinue (/home/(user)/undexer/node_modules/.pnpm/pg@8.12.0/node_modules/pg/lib/client.js:272:18) at Connection.emit (node:events:519:28) at /home/(user)/undexer/node_modules/.pnpm/pg@8.12.0/node_modules/pg/lib/connection.js:117:12 at Parser.parse (/home/(user)/undexer/node_modules/.pnpm/pg-protocol@1.6.1/node_modules/pg-protocol/dist/parser.js:36:17) at Socket. (/home/(user)/undexer/node_modules/.pnpm/pg-protocol@1.6.1/node_modules/pg-protocol/dist/index.js:11:42) at Socket.emit (node:events:519:28) at addChunk (node:internal/streams/readable:559:12) at readableAddChunkPushByteMode (node:internal/streams/readable:510:3) at Readable.push (node:internal/streams/readable:390:5), original: Error: SASL: SCRAM-SERVER-FIRST-MESSAGE: client password must be a string at Object.continueSession (/home/(user)/undexer/node_modules/.pnpm/pg@8.12.0/node_modules/pg/lib/crypto/sasl.js:24:11) at Client._handleAuthSASLContinue (/home/(user)/undexer/node_modules/.pnpm/pg@8.12.0/node_modules/pg/lib/client.js:272:18) at Connection.emit (node:events:519:28) at /home/(user)/undexer/node_modules/.pnpm/pg@8.12.0/node_modules/pg/lib/connection.js:117:12 at Parser.parse (/home/(user)/undexer/node_modules/.pnpm/pg-protocol@1.6.1/node_modules/pg-protocol/dist/parser.js:36:17) at Socket. (/home/(user)/undexer/node_modules/.pnpm/pg-protocol@1.6.1/node_modules/pg-protocol/dist/index.js:11:42) at Socket.emit (node:events:519:28) at addChunk (node:internal/streams/readable:559:12) at readableAddChunkPushByteMode (node:internal/streams/readable:510:3) at Readable.push (node:internal/streams/readable:390:5) }

Node.js v20.15.0

(I see the error doesn't seem to like my password, but it's all set as in env.example in .env - I assume .env is the right filename?)

[opsecx]

Any idea?

[egasimus]

That is strange, it should be picking it up. I've just moved the import "dotenv/config" statement to src/config.js, and made that file executable. So, now if you run src/config.js by itself you should be able to see whether it's picking up your environment variables.

(I've also updated the @fadroma/namada dependency to build again Namada 0.41, but haven't had the opportunity to test it further than that.)

[opsecx]

That is strange, it should be picking it up. I've just moved the import "dotenv/config" statement to src/config.js, and made that file executable. So, now if you run src/config.js by itself you should be able to see whether it's picking up your environment variables.

(I've also updated the @fadroma/namada dependency to build again Namada 0.41, but haven't had the opportunity to test it further than that.)

define "run" in the context of a non-executable? nodejs file.js?

assume I should repull? (need to rebuild as well?)

[opsecx]

pretty sure I am just using the ordinary format for the postgres url so no idea what's going on..

[egasimus]

node src/config.js, though I've also made it executable using #!/usr/bin/env node. You need to repull to get the latest changes. You don't need to rebuild to check the config, but you will need to rebuild to enable the 0.41 update.

[opsecx]

node src/config.js, though I've also made it executable using #!/usr/bin/env node. You need to repull to get the latest changes. You don't need to rebuild to check the config, but you will need to rebuild to enable the 0.41 update.

What exactly am I looking for in terms of results/outputs?

[egasimus]

In the output of src/config.js, you should be seeing the DATABASE_URL you specified in .env, as opposed to the default one or some invalid value.

[opsecx]

In the output of src/config.js, you should be seeing the DATABASE_URL you specified in .env, as opposed to the default one or some invalid value.

Ok so... node src/config.js yields a series of configuration values, amongst others correctly reproducing the database url string I put in .env.

When running ./undexer index, I get the same error as before (ConnectionError [SequelizeConnectionError]: SASL: SCRAM-SERVER-FIRST-MESSAGE: client password must be a string)

Halp.

[opsecx]

Btw I reinstalled everything from scratch and recloned repo, so nothing stale lying around.

[opsecx]

The entry has this format btw: DATABASE_URL=postgresql://localhost/actualdbname?user=actualusername&password=actualpassword (on a single line even if there are some display issues here)

[opsecx]

Ok, fixed it. Your env.example needs updating to a different database url format.

The following works:

DATABASE_URL=postgres://username:password@0.0.0.0:5432/dbname

(for standard postgres port)

Please update.

[opsecx]

Just putting here what I think needs to be in Readme for a non-docker setup:

Prerequisites:

Nodejs Rust (install via rustup) Rust WASM toolchain (cargo install wasm-pack) pnpm package manager: npm install -g pnpm

Installation/configuration:

git clone (undexer repo) copy env.example to .env and put in correct values (if no post/pre put same rpc in both fields) git submodule update --init --recursive pnpm i cd fadroma/packages/namada && npm run build:wasm:dev ./undexer index (from repo root)

[egasimus]

Updated docs and defaults. Thank you so much for your input! Please feel free to reopen if you find anything else to be missing.