Migrating to subgraph studio

aminlatifi commented 1 year ago

The subgraph team is going to sunset hosted service (the one we are using now), and push customers to use the decentralized subgraph paradigm, studio.

They say the studio is ready for the Mainnet network, and they will support Gnosis-Chain and Goerli in close future. They will still support the hosted service for former chains till they support them completely.

I tried to launch our stage Goerli instance on the decentralized subgraph, but I wasn't successful and got blocked as they don't support it completely.

BTW, we must be ready to migrate our Mainnet deployment to the studio. These items we must consider:

Deployment needs on chain transaction and signaling
API is not free, we must create APIs, and restrict them to our hosts (giveth.io) and connect to our own subgraphs

mhmdksh commented 1 year ago

Hey @aminlatifi any specific dates on when they will sunset their hosted service??

aminlatifi commented 1 year ago

Hey @aminlatifi any specific dates on when they will sunset their hosted service??

They say Q1 2023, https://thegraph.com/blog/sunsetting-hosted-service/

aminlatifi commented 1 year ago

@geleeroyale @mhmdksh Can we lean on the devops team to manage the deployment and balance management on the decentralized subgraph (studio)? I made this PR and tested deployment on the studio env. When it become merged and be tested, no further development will be needed.

geleeroyale commented 1 year ago

Of course. We will handle it. Setup might need developer guidance, but we can keep it running for sure 👍

aminlatifi commented 1 year ago

Thanks @geleeroyale The code to be deployed is ready, it's deployed on the hosted services. The instruction to deploy on studio is documented well, just you need to choose a correct configuration, e.g. production-gnosis, production-mainnet, staging-gnosis, etc. for deploying each instance.

geleeroyale commented 1 year ago

Ok! Let me see how far I can get and then I'll ping you

geleeroyale commented 1 year ago

Update - All subgraphs have been deployed. They have not been published yet as we await a grant from The graph foundation to pay for the decentralized hosting. Also we are still awaiting subgraph studio support for Görli and Gnosis Chain to be able to publish the complete set.

divine-comedian commented 1 year ago

big technical debt we have to address of migrating from hosted service to subgraph studio - first attempt had failed - the queries were not resolving values, our DB was empty!!

We should figure this out soon

@aminlatifi says last resort if studio doesn't work we may have to set up our own subgraph services

geleeroyale commented 1 year ago

It would be great to have another go. If we are able to pinpoint our exact problems please do not forget that we have a direct line (chatroom) with representatives from TheGraph. Also its highly subsidized by them and we should at least use the tokens before we invest in our own infrastructure.

Having said that - we are down to look into self-hosting. But I fear the requirements might be heavy - its mentioned that server costs might clock in at around 350€. Not sure - but maybe we could increase performance by running our own indexer instead of directly hosting the subgraphs?

Here is some reading material on the topic - https://thegraph.com/docs/en/network/benefits/

divine-comedian commented 2 months ago

Amin and I had a discussion while looking at all the other possible options.

We should move forward with upgrading our subgraphs directly to The Graph Studiio, we still have GRT tokens to spend, and after that the service likely won't cost more than what we would spend with a similar service and can support all the chains we need now and future chains like polygon zkEVM.

Who can be made accountable to be sure this gets done in the next 2 weeks? @geleeroyale ??

aminlatifi commented 2 months ago

@geleeroyale Please update all our github actions to deploy to studio. There you can find the list of used subgraphs as well.

geleeroyale commented 2 months ago

I updated our github-actions to (also) deploy on TheGraph studio.

Some notes/caveats:

Optimism-Görli IS NOT SUPPORTED by TheGraph (see a list here -> https://thegraph.com/networks/) ~~- We seem to have no deployment manifest for Optimism production (I must be doing something wrong)~~ PR to add optimism (https://github.com/Giveth/giveconomy-subgraph/pull/73), I saw you had a PR lingering as well @aminlatifi - you can merge if you want
I left out the Görli deployment due to the network being deprecated
subgraphs are deployed for free and it works great from github actions
however, if only deployed (and not published) there is a limited amount of queries you can run against it as it is not part of the decentralized network - let's check but for staging only deploying might be ok
to publish a fee has to be paid to the smart contracts managing the decentralized network
the fee on Ethereum mainnet is at a gas price of 14GWEI(!) only and ETH currently about 3k 15 USD gas cost and another 43 GRT token cost (GRT is at 0.30 USD ~ 11 USD in fees total) === a sum total of about 26USD (we need to pay this PER subgraph we want to publish)
there is an option to migrate to Arbitrum for publishing (the indexing network is independent from the network to be indexed) which we should do in my opinion.

Edit: Publishing is a manual process that can't be part of GitHub actions unfortunately Edit2: We need some Arbitrum ETH and mainnet ETH from the DAO @divine-comedian to continue operating the devOPS multisig (a proposal has been made in the forum -> https://forum.giveth.io/t/top-up-the-devops-multisig/1441)

divine-comedian commented 2 months ago

Are you blocked by needing ETH? We can try to expedite this

Giveth / giveconomy-subgraph

Migrating to subgraph studio #38