we're compiling a JSON schema for every message, of which there can be thousands a second of potentially
The problem of passing schema as a property in a worker message is that for each message in lifecycle we never know what to expect and so we have to continuously parse newly received schema JSON formats
we could cache them into a global file, but I'm not sure this is pretty either
Another option would be to define them somewhat on a strategy level and then load them and compile them when the strategy is loaded. Then we could match by the commissioner. There are pros and cons for this. The pros are that this would further cement our ideal of 1 strategy = 1 data source, the cons are that sometimes we necessarily need two data sources to e.g. get a block number first before crawling all block logs
TimDaub
commented
1 year ago
schemaas a property in a worker message is that for each message in lifecycle we never know what to expect and so we have to continuously parse newly received schema JSON formats