search

vacuumlabs / adalite-backend-service

Icarus, a reference implementation for a lightweight wallet developed by the IOHK Engineering Team.
Apache License 2.0
2 stars 7 forks source link

[WIP] Cardano rest integration #86

Closed xdzurman closed 4 years ago

xdzurman commented 4 years ago

Replaces tangata-manu and its database by cardano-rest.

ksemaev commented 4 years ago

Can you please tell what would be that new setup? cardano-http-bridge will be replaced with https://github.com/input-output-hk/cardano-node + https://github.com/input-output-hk/cardano-db-sync ? Would we use extended db-sync mode (used by graphsql setup) or a basic mode (used by cardano-rest)?

xdzurman commented 4 years ago

Yes exactly, cardano-node along with cardano-db-sync. Currently we're running cardano-rest in docker-compose for development and testing. So far, we don't need the Epoch table from the extended db-sync for our queries.

For sending transactions, we'll be using cardano-submit-api for now. This leaves the cardano-explorer-api component the only redundant part of cardano-rest for us.

ksemaev commented 4 years ago

@xdzurman one more question: will we drop importer https://github.com/vacuumlabs/adalite-importer now, and backend would work with PG from cardano-db-sync?

xdzurman commented 4 years ago

@ksemaev Yes, adalite-importer, a.k.a tangata-manu will be dropped. New database schema from cardano-db-sync will be used. For now, for sending transactions, cardano-submit-api from cardano-rest will be used.

ksemaev commented 4 years ago

@xdzurman sorry for being annoying :) but can you please tell: 1) what will we place here https://github.com/vacuumlabs/adalite-backend-service/blob/cardano-rest-integration/docker/docker-compose.yml#L27 2) How will we connect backend to submit-api? Will we expose it separately maybe?

xdzurman commented 4 years ago

@ksemaev You're not annoying, feel free to ask anything :)

1.) I pushed the changes, this will depend only on your port configuration in .env 2.) the submit-api is composed along with other cardano-rest part, but I'm not sure if I understood the question

ksemaev commented 4 years ago

@xdzurman TY for response! Previously we have exposed only backend-api endpoint to connect to. Now we are also using submit-api. So the question is: will we connect to cardano-submit-api through adalite-backend-api (in this case only backend-api would be exposed for external connections), or will we connect to submit-api separately from backend-api (in this case there would be 2 endpoins exposed for external connection, also the setup will be changed compared with previous importer/backend setup) As I see from your commit - it's the first option, we will be exposing only backend api, and it will transfer calls to submit-api by itself. Please correct me if I'm wrong :)

xdzurman commented 4 years ago

@ksemaev Yes, you are correct :) Submit-api will be hidden (visible only to adalite-backend) and only communication with adalite backend will be necessarry.

jaydp17 commented 4 years ago

We're running the commit 373c84b but getting the error relation "tx_body" does not exist. Is there a way we can fix that problem?

xdzurman commented 4 years ago

@jaydp17 How do you run the app? I've replaced cardano-db-sync image in docker compose by the one which supports tx_bodies as well. Note that the tx_body extention is not officialy released by IOHK, but rather by Emurgo and also it does not collect all transaction bodies at any point in time, it only collects those it comes across. So if you're already synced and you start the extended version, it will only add tx_bodies of transactions that occured after the start. Therefore it's necessary to clear the database and restart the syncing process to get all tx_bodies. So if you clear the database, start cardano-rest and adalite-backend service by COMPOSE_PROJECT_NAME=<custom_prefix_to_containers> docker-compose up and get in sync again, it should work. We will document all needed steps shortly.

ksemaev commented 4 years ago

Not sure if it's expected, but first runs of migrations result (several restarts fix it):

migration file "20200629231126_create_block_time_index.js" failed
migration failed with error: create index "idx_block_time" on "block" using btree ("time") - deadlock detected
Process 20235 waits for ShareLock on relation 16448 of database 16401; blocked by process 20676.
Process 20676 waits for RowExclusiveLock on relation 16489 of database 16401; blocked by process 20235.
error: create index "idx_block_time" on "block" using btree ("time") - deadlock detected
    at Connection.parseE (/adalite/node_modules/pg/lib/connection.js:545:11)
    at Connection.parseMessage (/adalite/node_modules/pg/lib/connection.js:370:19)
    at Socket.<anonymous> (/adalite/node_modules/pg/lib/connection.js:113:22)
    at Socket.emit (events.js:193:13)
    at addChunk (_stream_readable.js:295:12)
    at readableAddChunk (_stream_readable.js:276:11)
    at Socket.Readable.push (_stream_readable.js:231:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:154:17)
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1
xdzurman commented 4 years ago

@ksemaev Thank you for pointing this out :) I managed to replicate the error after about 12 tries, I thought the error was in knex. However, the root cause of this is that cardano-db-sync starts inserting data into the database and sometimes, a deadlock might occur because some rows are frozen due to creation of indices from adalite-backend. We currently don't have an easy way to fix this, but we'll most likely create a fork of cardano-db-sync and move migrations there, so that they are run when the schema is built. We'll either just maintain this fork for us or make a pull request to official cardano-db-sync repo. For now, adalite being restarted on failure solves this, although not ideally.

Edit: these indices from migrations were merged into cardano-db-sync, so we will soon remove all migrations from adalite-backend-service altogether.