search

near / near-indexer-for-explorer

Watch NEAR network and store all the data from NEAR blockchain to PostgreSQL database
https://near-indexers.io/docs/projects/near-indexer-for-explorer
GNU General Public License v3.0
124 stars 56 forks source link

Add Dockerfile and .dockerignore #177

Closed mieubrisse closed 2 years ago

khorolets commented 2 years ago

Hey @mieubrisse! Thanks for the PR!

Could you clarify have you considered using multi-stage Docker builds?

I'll leave a few review comments soon.

mieubrisse commented 2 years ago

Hey @khorolets , yep, I actually started this Dockerfile using a multistage build and agree it'd be a great image space-saver. The issue I encountered is the diesel step which needs to run as part of startup, which requires the Rust toolchain for cargo install, which blows up the image size again. There's probably a way to get diesel with a much smaller image, but I wasn't focused on optimizing this image very much - just getting it to work as a v0.

khorolets commented 2 years ago

@mieubrisse yeah. My concern is that the current approach is a bit suboptimal for production use because migration step and --store-genesis are needed ideally only once. OK, the migration step might be needed from time to time, but still.

How about refactoring this to a multi-stage build? I believe we can find a ready-one Docker image with diesel-cli already and describe how to use it, what do you think?

frol commented 2 years ago

diesel step which needs to run as part of startup, which requires the Rust toolchain for cargo install

Can you share the specific error? I have tested it on my end and I can install diesel in a build image, and then run it on a fresh ubuntu image with libqp-dev package installed.

mieubrisse commented 2 years ago

@frol how did you do that? Just install libqp-dev and copy the diesel binary across?

mieubrisse commented 2 years ago

@khorolets yeah ideally this repo would produce two Docker images - one for the indexer and one based on the postgres image with the database already created - but that was the approach taken by the no-longer-functional https://github.com/near/local/blob/master/docker-compose.yml and the database Docker image fell into disrepair pretty quickly it seems. I opted to do the diesel migration as part of this main repo to ensure that things would be more likely to stay maintained/usable

frol commented 2 years ago

how did you do that? Just install libqp-dev and copy the diesel binary across?

Yep

mieubrisse commented 2 years ago

Any idea what the other packages besides libpq-dev are that are needed, @frol ? I can spend a little while giving a multistage build a go, but the build takes 45min so getting it wrong really hurts (though I tried to be clever and push the build as early in the Dockefile as possible)

From https://github.com/near/nearcore/blob/master/Dockerfile I also have libssl-dev and ca-certificates... any others that I need?

frol commented 2 years ago

@mieubrisse with multistage it should only build it once so once you have the second stage, it is just a matter of copying files over and quickly iterate on the missing libraries. I believe only libqp-dev is necessary, but I don't have data on me, so your exploration will be as good as mine.

mieubrisse commented 2 years ago

Updated to multistage build and verified that it works @khorolets @frol . Image is now 153 MB rather than 4.6GB