search

classic-terra / core

GO implementation of the Terra Classic Protocol
Other
65 stars 44 forks source link

Review Option for Multiple SDR's for Oracle Feeder #230

Closed LuncBurner closed 1 year ago

LuncBurner commented 1 year ago

Problem definition A.E suggested the use of multiple SDR's for use with the Oracle feeder to ensure price stability.

Acceptance criteria

LuncBurner commented 1 year ago

Previous versions of Oracle Feeder to compare against:

https://github.com/aeuser999/oracle-feeder

https://github.com/aeuser999/tr-oracle-feeder

Both versions derive from this commit from TFL's repo: git clone https://github.com/terra-money/oracle-feeder git checkout 5084f8441eb74f518163250ae38837bc2566aebc

https://github.com/terra-money/oracle-feeder/commit/5084f8441eb74f518163250ae38837bc2566aebc

aeuser999 commented 1 year ago

Thank you @LuncBurner .

Just adding this note that I mentioned previously in the TerraClassic Validator Discord:

You can see the sources here:

Fiat Providers (and SDR / XDR) Notes: I did find that, on looking at https://github.com/classic-terra/oracle-feeder/tree/main/price-server/src/provider/fiat/quoter that the other fiat sources are still in the code base.

I had wondered if just adding the fiat provider classes configuration into the default.js configuration file would work, and so I tested that (one single entry being tested in fallbackPriority per provider), but it does look like, in some cursory unit testing, that those fiat provider classes need reworking to work with the changes TFL made for Terra v2 (it switched the pair order around). I did come up with:

Crypto Providers Notes: I also found that some of the crypto providers in the current default-sample.js config file for the price-server are not working correctly. I did come up with:

When I tested with only a single fallbackPosition for each crypto provider (I am in a restricted zone for Binance, so I could not test that one), only CoinGecko worked (with the exception of the updated Kraken file). It is probably because the others do not have a LUNC/USD pair (although they have LUNC/USDT or LUNC/USDC or LUNC/BUSD - so they would need to be translated).

I am not sure if any of this is helpful, but wanted to share if helpful. Since it was just preliminary, unfinished, and tested in a cursory manner, that is the reason I just posted back links to the code mentioned (rather than a PR).

I hope you have a great day today :)

alchemist-ti commented 1 year ago

The difference between the previous version and the current version: https://github.com/aeuser999/tr-oracle-feeder/compare/5084f8441eb74f518163250ae38837bc2566aebc...main.

It mainly contains:

alchemist-ti commented 1 year ago

@LuncBurner Now classic terra uses https://github.com/classic-terra/oracle-feeder.

Availability of multiple SDR sources:

source URI available
ExchangeRate https://api.exchangerate.host/latest?USD&symbols=XDR,KRW,MNT,EUR,CNY,JPY,GBP,INR,CAD,CHF,HKD,AUD,SGD,THB,SEK,DKK,IDR,PHP,MYR,NOK,TWD true
Frankfurt https://api.frankfurter.APP/latest?from=SDR&to=USD false
fer.ee https://api.fer.ee/latest?base=SDR&to=USD false
alchemist-ti commented 1 year ago

@aeuser999 Do you know of any other sources of SDR rates?

aeuser999 commented 1 year ago

The difference between the previous version and the current version: aeuser999/tr-oracle-feeder@5084f84...main.

It mainly contains:

  • Add DockerFile
  • Update dependencies: axios & @terra-money/terra.js
  • Update keys encrypt and decrypt
  • Refactor configs

Hi @alchemist-ti ,

I would add in the list that the older TR oracle version has the SHA256 fix to the keystore (I think this may be included in what you mentioned "Update keys encrypt and decrypt"):

It was one of the enhancements with that particular version.

Note: Also, I would add, that TFL did repurpose the oracle-feeder for Terra v2 purposes after commit 402996fd6a2c4078303f25c13bf0c8909bbbe02d in their repository, and reworked a number of things afterwards for their purposes. Those changes were the base version that the L1 JTF's version started from. Just pointing that out if helpful, if not that is ok too - just disregard (it does look like some of the fixes that Vinh made have adjusted these changes to work with Terra v1, while also retaining the L1 JTF SHA256 and other changes).

aeuser999 commented 1 year ago

@aeuser999 Do you know of any other sources of SDR rates?

Hi @alchemist-ti ,

Thoughts

I do notice that the following two sources do not give back SDR (or XDR - XDR being the way foreign exchange price sources will most likely list SDR): Source URL Response
Frankfurter https://api.frankfurter.APP/latest?from=SDR&to=USD {"message":"not found"}
fer.ee https://api.fer.ee/latest?base=SDR&to=USD < No Response Returned >

  For SDR/XDR there are those with the prior code still in the code base (that are listed in the old TR version's default-sample.js), such as: 'currencylayer', 'alphavantage', 'fixer', 'exchangerate' (bandprotocol stopped supporting SDR/XDR it appears, and the api endpoint listed in the old TR code for bandprotocol does not work for me)(I notice bandprotocol was removed from the newer TFL and L1JTF version).

Fixer has had issues with MYR, and so it may be good if we can find other sources to use than Fixer. Or, maybe filter out MYR for Fixer in default-sample.js, similar to how SDR is filtered out from Frankfurter and Fer, so that another sources prices for MYR, rather than Fixer's, prevail. Or maybe an even better fix could be to update the logic so that if one source's price for an asset are more than 50% off [as a suggested % of a wild variation] from other sources, then filter out that source's price for the asset from consideration, and instead consider instead the next sources' price in the fallback order for that asset, with the change to do this implemented in the code itself (ie. move to the next source that has pricing in the priority list).

Also, I don't know if you saw this already, just passing it along if helpful (disregard if not), SDR from default-sample.js is translated to XDR in the code for sources that list it as XDR, and then translates XDR back to SDR after pricing (so that it will work to use SDR in the default-sample.js).

Answer

But, to answer your direct question "Do you know of any other sources of SDR rates", the problem is coming up with free, yet reliable, sources for SDR / XDR, that update fiat prices regularly. Here are some providers, in addition to the ones listed above, that have SDR / XDR (although paid - but do have a free subscription for testing purposes with rate limit):

I hope that helps out a little bit, and that you have a great day today :)

LuncBurner commented 1 year ago

@aeuser999 @alchemist-ti Excellent collaboration. If we need to pay a small monthly fee to a site that produces another source for SDR and it is advantageous to us, I would think we could go ahead and purchase a 2-year subscription to such a service.

aeuser999 commented 1 year ago

@aeuser999 @alchemist-ti Excellent collaboration. If we need to pay a small monthly fee to a site that produces another source for SDR and it is advantageous to us, I would think we could go ahead and purchase a 2-year subscription to such a service.

Hi @LuncBurner ,

That sounds awesome. I may be missing something (if so, just disregard), but the cost has traditionally been for validators to pay at signup (at least, it is mentioned in earlier versions of the default-sample.js file, and traditionally validators have picked up their own subscription costs). For instance CurrencyLayer (Business), AlphaVantage (Premium), and Fixer (Professional) each have a subscription cost to provide an appropriate amount of requests per month without rate limiting. The newer ones I mentioned above have the same types of plans for enough requests per month.

Each of the API services have a free plan that should work for testing (normally 500 or so requests), although you are right that more extensive testing may require a paid plan to make sure not to hit the limit on requests in testing.

As a thought: Even if a validator picked at least 3 good paid sources (to add to the free ExchangeRate, and other non-SDR free fiat sources), it would give a good variety for which 3 sources they can pick from (if we had each of these fiat providers - reworking those that still exist in the code base, and then possibly using those classes as a template to add in the new fiat providers?). Just a thought though...

Don't know if that helps (again, just disregard if not, or if I misunderstood).

I hope you are having an awesome day today :)

alchemist-ti commented 1 year ago

@aeuser999 thanks for your detailed elaboration.

As a thought: Even if a validator picked at least 3 good paid sources (to add to the free ExchangeRate, and other non-SDR free fiat sources), it would give a good variety for which 3 sources they can pick from (if we had each of these fiat providers - reworking those that still exist in the code base, and then possibly using those classes as a template to add in the new fiat providers?). Just a thought though...

That's good advice.

As you comment here https://github.com/classic-terra/core/issues/230#issuecomment-1565535456, currencylayer, alphavantage, fixer are also available now. Seems we can revert some code to support them quickly, and the traditionally validators can use them via the previous API keys.

In the long run, we can support more sources you mentioned.

alchemist-ti commented 1 year ago

@aeuser999 I re-supported currencylayer and fixer in https://github.com/classic-terra/oracle-feeder/pull/16, can you help review it?

aeuser999 commented 1 year ago

@aeuser999 I re-supported currencylayer and fixer in classic-terra/oracle-feeder#16, can you help review it?

@alchemist-ti Sorry, I have not had the chance to be able to do this yet. I will try to make some time on Saturday or Monday to look at it. Thank you so much for all your work in adding those - that is awesome :)