Support custom RPC url(s)

[valiafetisov]

Goal

The user is able to supply RPC url via environment variable instead of infura project id

Context

1. Currently we only let users (who wants to start our UI or the bot locally) provide INFURA_PROJECT_ID. But this is limiting, since users may use different providers as well as custom / local nodes
2. Currently the user is only able to connect to a set of networks from the list of supported ones and there is no possibility to dynamically add new networks to our UI
   • Use case: user is able to use custom PRC endpoint of their own
   • Use case: we're able to integrate chaoslabs forks

Note: most probably, those two scenarios require two different solutions. This issue outlines them here because we don't want them to be in conflict with each other.

Tasks

• [x] Come up with a solution(s), outline it here
• [x] Discuss
• [x] Implement

[valiafetisov]

Just thinking out loud atm:

There are several problems which needs to be taken into account when switching to custom RPC_URL instead of INFURA_PROJECT_ID:

• Our frontend supports different networks, while one RPC url provides access to a single network, therefore:
  • A in case the provider is known (ie infura) we can extrapolate the other urls from it, otherwise disable network selector
  • B we can disable network selection right away, if RPC_URL instead of INFURA_PROJECT_ID is provided
  • C we can add this network as additional / custom RPC instead
• Technically, in order to use custom RPC url, we need to know its chainId, but we can also fetch it from the endpoint
  • Currently we also have ETHEREUM_NETWORK that can be set in the bot, which should be obsolete in case when RPC_URL is set, so we may want to get rid of it
• RPC_URL should be applied at runtime and there should be new method to add new networks (at least on the frontend) so that adding new simulation networks is possible in https://github.com/sidestream-tech/unified-auctions-ui/issues/83

[valiafetisov]

So the proposal would be to:

• add RPC_URL env variable
• remove INFURA_PROJECT_ID
• remove ETHEREUM_NETWORK and derive this value from the RPC_URL (by making RPC call for chainId)
• detect if RPC_URL is infura (eg using regex), get project id from the url and support multiply networks automatically
• store networks in modifiable global state, instead of a static config (or rather in addition to that)
  • create a method to add new networks
  • on page load, add one of several networks (based on the detection described above)

[zoey-kaiser]

I had one additional question. We currently have a etherscanURL, assigned for every Network. How would we want to handle this if a rpcURL is assigned, with no matching etherscanURL (+ how would we know if one is accessible or not)?

I would recommend the following solution:

• We only support the links to transactions, if a infura rpcURL is provided. As we already parse the url to identify if it is a Infura URL, we can then reinject the etherscan urls.
• If an external rpcURL is used, we deactivate all the links (or find another place to refer the user to?)

I would love to hear some of your thoughts on how to handle this! 😄

[zoey-kaiser]

Alternatively,

we could include a second environment variable, called etherscan link. If this is set to false or undefined, the links are disabled.

[valiafetisov]

We currently have a etherscanURL, assigned for every Network. How would we want to handle this if a rpcURL is assigned

Good question. I imagine that we can just derive network name from chainId (fetched from the RPC) and construct urls based on that. In case of the testing environment, when id will be the same as the main network, the url will be invalid, but the developer should be aware of that. Adding support for block explorers that work with localhost can be added in another issue and is out of scope of this one.

[zoey-kaiser]

Good question. I imagine that we can just derive network name from chainId (fetched from the RPC) and construct urls based on that. In case of the testing environment, when id will be the same as the main network, the url will be invalid, but the developer should be aware of that. Adding support for block explorers that work with localhost can be added in another issue and is out of scope of this one.

Sounds like a plan! I should easily be able to derive it from the name and then I can open a new issue, once this PR is merged for adding custom block explores. But I also agree, that this does not have the highest priority and developers, hosting the site via their own custom RPC or networks, should be able to manage this issue for themselves until we get to it!

[zoey-kaiser]

After further discussion in a call we settled on the following system:

• Create a dynamic provider/signer setup
  • You can add and remove providers/signers after first load (allows us to add simulation setups)
• On first load the RPC Url is parsed:
  • It is checked if the URL contains an Infura Project ID. If it does:
  • Three networks are automatically added based on Infura links: mainnet, kovan, goelri
  • If not the single RPC url is added. A call is made to retrieve the ChainID. We check if the ChainID is linked to a network name and use that name as the display name. If no name is found we use the ChainID as the display name (Later PR could add support to dynamically get all networks and ChainIDs)
• All RPC URLs are stored and can be retrieved after wards
• The network dropdown is based on this dynamic provider/signer system, allowing users to switch back and forth when new URLS are added.

Note: Only infura project IDs will generate the correct EtherScan links. If you use a network not supported by Etherscan, the links will point to the mainnet browser, which will be missing your data. However until we get to it, developers will need to manage this bythemselves.

[valiafetisov]

Thanks for the summary!

You can add and remove providers/signers after first load (allows us to add simulation setups)

I think provides/signers are different from networks – you may have 3 networks, where only one of them is initialized (ie have provider). So the idea is to have networks global state similar to providers/signers (eg with setNetwork function which adds new network to the list based on PRC url alone)

On first load the RPC Url is parsed

Not necessarily on the first load setNetwork can be called later on, on the first load we parse RPC_URL env variable, but the same steps of detecting project id etc can be applied on every setNetwork call

Everything else sounds good ✨

[zoey-kaiser]

I had a question about how to manage/sync the network store on the frontend. I wanted to make the network dropdown be based on the networks global variable defined in the core. How can I ensure, that the UI is synced and updated when a new Network is added. In theory I need to use a VueX store for this, but is there a way to sync said store with the core functions?

Maybe there is another alternative approach I am missing, that I currently face the issue, that I update the network value in from the auctions-core, but the changed values are not reflected in the frontend.

Edit: I am also having issues, with having the bot reflect the updated changes to the network global store.

[valiafetisov]

One possible approach is to emit network changes. So the setNetwork can not only modify global state, but re-emit the whole network config via subscribeToNetworks(callback) – which is then can be used in the store:

// inside to the core
const callbacks = [];
const subscribeToNetworks = function (callback) {
    callbacks.push(callback);
}
const setNetwork = function () {
    // the new function that adds a network
    callbacks.forEach(callback => callback(allNetworks));
};

// inside to the store
export const actions = {
    // ...
    setup({ rootState, dispatch }: ActionContext<State, any>) {
        subscribeToNetworks(() => {
            // modify state 
        });
    },
};

[zoey-kaiser]

The next issue I am running into is how to define the currently selected Network. I was thinking of storing this variable, just like all the networks. I would then also setup a subscription service, that the frontend can callback to this.

This would remove the need for us to pass network too call core functions, as the selectedNetwork would be retrieved from the core. I think I would keep the network prop, but make it optional, so that one could in theory override and use a different network, from the one you are currently on.

Does this make sense or should I approach it in a different way?

[valiafetisov]

how to define the currently selected Network

I think this doesn't need to be stored in the core. And it's already stored in the network vuex store. As we discussed earlier, refactoring network parameters is out of scope right now.

[zoey-kaiser]

how to define the currently selected Network

I think this doesn't need to be stored in the core. And it's already stored in the network vuex store. As we discussed earlier, refactoring network parameters is out of scope right now.

The bigger issue is the bot. As we want to remove the network parameter for this, I am not too sure how to store it there. Should I just make a variable and then set the network during the bot setup?

[valiafetisov]

One possible approach is to emit network changes

Another one is that setNetwork can simply return newly added network config.

As we want to remove the network parameter for this, I am not too sure how to store it there.

Technically, you don't need to store it, since you can derive it from the RPC_URL. You can simply get the first of the added network either via const config = await setNetwork(RPC_URL) (if you call setNetwork from within the bot) or by using subscribeToNetworks from above if you've already implemented it