Setup continuous deployment

frol commented 3 years ago

Here are the problems we currently have:

[x] There is no automation around provisioning and auto-start for the Indexer nodes, so we manually configure and maintain the nodes uptime
[x] Sometimes indexer nodes fall off sync process due to protocol upgrade to the version that is not supported by the nearcore we pulled in as a dependency

Currently, we run Indexer for Explorer manually on GCP nodes under tmux session, and we bump nearcore version once in a while mostly to a recent nearcore commit version instead of using a proper release to match the rest of the network.

We need to follow the same provisioning strategy as we have for archival RPC nodes, and if we automate this process, we should be able to manually bump the version of nearcore in the Cargo.toml, go through CI, merge to master, and deploy the new release to the nodes. Ideally, we should use separate releases for testnet, mainnet, and betanet deployments.

P.S. we don't have betanet deployment at all, so we can start with it to debug any issues on the way (it will unblock https://github.com/near/near-wallet/issues/1510)

stefanopepe commented 3 years ago

Any updates on this issue? Betanet users are still blocked by this one

chefsale commented 3 years ago

@jimmy3dita @frol not sure how is this related to betanet users?

stefanopepe commented 3 years ago

@jimmy3dita @frol not sure how is this related to betanet users?

as of today, users can't use https://wallet.betanet.near.org.

The outcomes are:

We have the indexer and the rpc available for the wallet
We disable entirely the possibility to use the wallet on betanet (which seems to me easier and best suited for the current betanet scope)

frol commented 3 years ago

@chefsale Betanet is a great place to test out our automation, though it will be quite involved due to the betanet nature (frequent wipes and use of nightly nearcore features):

[ ] we need some automation to build indexer-for-explorer following betanet releases (update the commit hash in Cargo.toml and build a new binary)
[ ] we need to wipe betanet database whenever there is hard-reset on betanet

@jimmy3dita How critical is that to have Wallet on betanet?

stefanopepe commented 3 years ago

@jimmy3dita How critical is that to have Wallet on betanet?

close to zero =)

frol commented 3 years ago

@chefsale I vote for closing this issue once we have automation of the deployment process and extract the rest of the items (nearcore releases tracking and hard-reset automation) into separate issues. WDYT?

khorolets commented 3 years ago

@frol I totally agree about closing.

Also wanted to add, that nearcore might include some changes in dependencies or structure changes (sometimes) that's not followed by Indexer for Explorer immediately and having Indexer for Explorer automatically built for betanet may end up in the situation where Indexer for Explorer is unable to be built.

We had some issues in the past when we had to update the logic or dependencies on the indexer side to build with newer version of nearcore. So I don't believe we can rely on automation for betanet purpose.

chefsale commented 3 years ago

Yes, agree on closing. We have automation for testnet and mainnet now fully in place. The only thing which is in question right now is betanet from this issue? We can discuss this seperately.

chefsale commented 3 years ago

Created an issue for indexer on betanet: https://github.com/near/near-indexer-for-explorer/issues/113

near / near-indexer-for-explorer

Setup continuous deployment #75