USD Value for old donations

[MoeNick]

@mohammadranjbarz Mateo added Giv to monoswap yesterday so new donations should work now. However for old ones we must write the script, the script doesn't exist.

This will also be specially helpful for tokens that don't appear on monoswap, since we allow them now.

I lookup with Mateo some apis like coingecko (but this one is too limited, gotta pay to use effectively). But we should probably look a varios options that would work side by side monoswap library, and the script would run everyday to update those values.

Didn't write the script because we were unsure which api to use.

Thanks Carlos

@MoeNick Should we create another story for what should we do for older donations? ( As Carlos said, I think it's not an easy or routine job, so it might need a little discussion as well)

Originally posted by @mohammadranjbarz in https://github.com/Giveth/impact-graph/issues/276#issuecomment-1001190043

[MoeNick]

Carlos comment

Mateo added Giv to monoswap yesterday so new donations should work now. However for old ones we must write the script, the script doesn't exist.

This will also be specially helpful for tokens that don't appear on monoswap, since we allow them now.

I lookup with Mateo some apis like coingecko (but this one is too limited, gotta pay to use effectively). But we should probably look a varios options that would work side by side monoswap library, and the script would run everyday to update those values.

Didn't write the script because we were unsure which api to use.

[mohammadranjbarz]

Carlos comment

Mateo added Giv to monoswap yesterday so new donations should work now. However for old ones we must write the script, the script doesn't exist.

This will also be specially helpful for tokens that don't appear on monoswap, since we allow them now.

I lookup with Mateo some apis like coingecko (but this one is too limited, gotta pay to use effectively). But we should probably look a varios options that would work side by side monoswap library, and the script would run everyday to update those values.

Didn't write the script because we were unsure which api to use.

I think we may change the title to USD Value for old donations because it's not limited to GIV donations, it may happen to all donations

[MoeNick]

It's open for discussion guys, @mohammadranjbarz would u pls list our choices, what we should do?

[CarlosQ96]

Honestly I think that for tokens that we don't support we should run a script to update them with other apis.

1. What service should we use? (In case it can't be added to our monoswap library).
2. When should the script run? (to avoid a price difference when we insert the values, OR I think we with the date with can get the price in a certain moment using the creation Date)

I double checked Coingecko with Mateo, free plan is 50 request in 1 minute. If we want more we gotta pay. There should be other services but gotta investigate.

In the case of GIV, that was our mistake we forgot to add it to the monoswap instantly, so the script should also update old donations with GIV with monoswap succesfully.

[mohammadranjbarz]

Honestly I think that for tokens that we don't support we should run a script to update them with other apis.

1. What service should we use? (In case it can't be added to our monoswap library).
2. When should the script run? (to avoid a price difference when we insert the values, OR I think we with the date with can get the price in a certain moment using the creation Date)

I double checked Coingecko with Mateo, free plan is 50 request in 1 minute. If we want more we gotta pay. There should be other services but gotta investigate.

In the case of GIV, that was our mistake we forgot to add it to the monoswap instantly, so the script should also update old donations with GIV with monoswap succesfully.

Thanks Carlos Because the price is very important ( we calculate givback based on donation price) we should be sure that every donation has price, so I think this script is almost necessary to make sure that no donation is missed, but as you said we should check it to how to do it, does monoswap support price for othertimes or just it get us price for Now time? If we can use it in the cronjob script, it would be great, otherwise we should use something like coingecko ( And I think 50 request per minutes is enough)

[CarlosQ96]

Monoswap only returns the current price. Coingecko can return the price history at a given date. HOWEVER, the issue is coingecko uses their internal ids to request tokens, we have to export first their token-ids to properly use the APIs thats the heavy part. (4k+ tokens) The file is huge, and even breaks their website hahaha Maybe we should export it to csv or something.

But after initial export that you can consult by history just fine.

Url: https://www.coingecko.com/es/api/documentation @mohammadranjbarz

[mohammadranjbarz]

Monoswap only returns the current price. Coingecko can return the price history at a given date. HOWEVER, the issue is coingecko uses their internal ids to request tokens, we have to export first their token-ids to properly use the APIs thats the heavy part. (4k+ tokens) The file is huge, and even breaks their website hahaha Maybe we should export it to csv or something.

But after initial export that you can consult by history just fine.

Url: https://www.coingecko.com/es/api/documentation @mohammadranjbarz

@MoeNick Do we support givbacks for all tokens are we just have a whitelist? because for verifying donation I just check the token that are in our whitelist, So @CarlosQ96 You can add coeingeckId of those donations to this list

https://github.com/Giveth/impact-graph/blob/staging/utils/tokenUtils.ts

that's not a long list and you can do it manually

[CarlosQ96]

@mateodaza @divine-comedian @MoeNick There is one issue, we also now accept all tokens (erc20 at least). Most probably they aren't on our whitelist. Should our script also try to update their usd/eth values? They can be some random obscure/new token haha

I think on the verification part, Mohammad would need to do something too.

[divine-comedian]

it seems like since we're accepting any erc-20 token now the overhead of mainting the monoswap repo might become too time consuming - I think the idea of using the coingecko API will save us a lot of headaches going forward - let's see how much it will cost

• the big problem is just as mohammad said that if we return no price for a donated token that has value or inaccurate price data there will be hell to pay if users begin to feel cheated.

[mateodaza]

@mateodaza @divine-comedian @MoeNick There is one issue, we also now accept all tokens (erc20 at least). Most probably they aren't on our whitelist. Should our script also try to update their usd/eth values? They can be some random obscure/new token haha

I think on the verification part, Mohammad would need to do something too.

I'd say if the price is found we save it, even if we do mistakes (let's say this obscure token is called the same as many)

We could run the script every 6 hours (or another rate) and check the price of missing tokens using the coingecko api, if the amount of tokens that we have to get price is greater than the api limit we put it on a job for a minute later.

The steps could be something like

• Get all donations from a particular date (this is important as price must be date related) that need to get price right
• Get all tokens from those donations
• Check and save price from these tokens (once)
• Update donations with given price

First time running this script may be costly and could take a while historically. But if we set it up right it it may be just fine as this solution assumes we won't get many "obscure" "price unknown" tokens in a day

[divine-comedian]

so this would still use monoswap and with any gaps in price data we fetch it from coingecko? and if we meter it out with a timed job as you say we wouldn't reach the rate limit.

does this solution have any limitations if transaction volume on Giveth increases greatly? (10x)

[mohammadranjbarz]

so this would still use monoswap and with any gaps in price data we fetch it from coingecko? and if we meter it out with a timed job as you say we wouldn't reach the rate limit.

does this solution have any limitations if transaction volume on Giveth increases greatly? (10x)

Just we are not sure that history of prices are precise in coingecko ( you can test GIV price for few days ago, to see the prices are not accurate)

[MoeNick]

Let's focus on this: How we can make sure we have a precise price history for missing transactions.

[WhyldWanderer]

I remember that this tool was created by Giveth Social Coding team in the past... it could be helpful. https://giveth.github.io/xchange-rates

[MoeNick]

Would u pls update this issue guys?

[mohammadranjbarz]

Would u pls update this issue guys?

It seems all of old donations have usdValue

[MoeNick]

We missed some GIV 2 trx from main net .

[divine-comedian]

how can this issue be tested?

[mohammadranjbarz]

how can this issue be tested?

There should not be any GIV donation that doesn't have usdValue and if you see in this screenshot the query return zero donations, so I think we can close this issue

[divine-comedian]

okay better question - how can I test this issue? Or how can it be moved to done?

[mohammadranjbarz]

okay better question - how can I test this issue? Or how can it be moved to done?

IMO you can move it to done, because the above query indicates that it works