dOrgJelli
commented
2 years ago Oh wow, this is really clever, love the "tags" functionality in option 2, and the namespacing in option 3 :clap:
Closed Niraj-Kamdar closed 2 years ago
dOrgJelli
commented
2 years ago Oh wow, this is really clever, love the "tags" functionality in option 2, and the namespacing in option 3 :clap:
dOrgJelli
commented
2 years ago A minor modification of option 3:
{
"api": "ens/testnet/defisdk.eth",
"recipes": {
"mainnet": {
"aave": {
"aUSDC": [
{
"query": "./mainnet.graphql",
"variables": {
"address": "0x9bA00D6856a4eDF4665BcA2C2309936572473B7E",
"network": "MAINNET"
}
}
],
"aDAI": [
{
"query": "./mainnet.graphql",
"variables": {
"address": "0x028171bca77440897b824ca71d1c56cac55b68a3",
"network": "MAINNET"
}
}
]
}
},
"polygon": {
"aave": {
"aUSDC": [
{
"query": "./mainnet.graphql",
"variables": {
"address": "0x9bA00D6856a4eDF4665BcA2C2309936572473B7E",
"network": "MATIC"
}
}
],
"aDAI": [
{
"query": "./mainnet.graphql",
"variables": {
"address": "0x028171bca77440897b824ca71d1c56cac55b68a3",
"network": "MATIC"
}
}
]
}
}
}
}w3 query ./recipe.json --namesapce mainnet.aave
Notes:
Niraj-Kamdar
commented
2 years ago We can do something similar to what we do for hardhat deploy scripts:
{
"api": "ens/testnet/defisdk.eth",
"recipes": {
"00_mainnet": {
"00_yearn": {
"00_yUSDC": [
{
"query": "./mainnet.graphql",
"variables": {
"address": "0x9bA00D6856a4eDF4665BcA2C2309936572473B7E",
"network": "MAINNET"
}
}
],
"01_yDAI": [
{
"query": "./mainnet.graphql",
"variables": {
"address": "0x028171bca77440897b824ca71d1c56cac55b68a3",
"network": "MAINNET"
}
}
]
},
"01_aave": {
"00_aUSDC": [
{
"query": "./mainnet.graphql",
"variables": {
"address": "0x9bA00D6856a4eDF4665BcA2C2309936572473B7E",
"network": "MAINNET"
}
}
],
"01_aDAI": [
{
"query": "./mainnet.graphql",
"variables": {
"address": "0x028171bca77440897b824ca71d1c56cac55b68a3",
"network": "MAINNET"
}
}
]
}
},
"01_polygon": {
"00_aave": {
"00_aUSDC": [
{
"query": "./mainnet.graphql",
"variables": {
"address": "0x9bA00D6856a4eDF4665BcA2C2309936572473B7E",
"network": "MATIC"
}
}
],
"01_aDAI": [
{
"query": "./mainnet.graphql",
"variables": {
"address": "0x028171bca77440897b824ca71d1c56cac55b68a3",
"network": "MATIC"
}
}
]
}
}
}
}With this we can just sort the keys and get the order of execution.With this we can even preserve order of execution for the child namespaces, not just the final queries:
Ex: npx w3 query ./recipe.json --namespace mainnet would first run yearn then aave namespace.
dOrgJelli
commented
2 years ago Really great points @Niraj-Kamdar :D
pwvpwvpwv
commented
2 years ago Agree with your modification of option # 3, @dOrgJelli. Just wanted to toss my 2¢ in on the problem of running specific recipes:
Traditional Java-style scoping and namespacing makes sense since people are familiar with it and it fits well into the JSON format. Passing a `-separated list of.`-separated namespaces to the CLI should prove to be a clean and simple way to specify several recipes to be run in a particular order.
In this recipe/menu/cookbook ontology, we are able to describe rather complicated instructions and steps without having to describe the variables or state graph explicitly within the recipes themselves. This presents the advantage of having simple, context-independent lists of "commands" that can be given to the CLI, or stored in the cookbooks themselves and executed later (or even some combination of the two). However, the downside here, obviously, is that that same context-independence does not afford sufficient complexity to allow for passing variables around, piping inputs to outputs, or otherwise working in a functional style with multiple recipes (for reference, this is more or less okay for systems like Chef where pure functions are less important than effectful ones, but this may not be the case for the kinds of queries that people might like to use, especially ones that depend on the output of previous queries in the chain).
One possible approach to inject context dependence would be to allow the recipes to bind variables to reference names (e.g. $mainnet.aave.aUSDC[0] might bind to the output of the first recipe in that namespace, assuming it was run and the result is known). This, however, increases the complexity immensly and begins to blur the line between programming language and configuration definition. Personally, I tend towards the camp of thinking that considers Turing-completeness in configuration languages a downside rather than a boon, so I believe that this approach would likely take us further away from where we actually want to go (per the above comments): a simple, first-phase solution to the problem as it currently exists.
{
"api": "ens/testnet/defisdk.eth",
"recipes": {
"mainnet": {
"aave": {
"aUSDC": [
{
"query": "./mainnet.graphql",
"variables": {
"address": "0x9bA00D6856a4eDF4665BcA2C2309936572473B7E",
"network": "MAINNET"
}
}
],
"aDAI": [
{
"query": "./mainnet.graphql",
"variables": {
"address": "0x028171bca77440897b824ca71d1c56cac55b68a3",
"network": "MAINNET"
}
}
]
}
},
"polygon": {
"aave": {
"aUSDC": [
{
"query": "./mainnet.graphql",
"variables": {
"address": "0x9bA00D6856a4eDF4665BcA2C2309936572473B7E",
"network": "MATIC"
}
}
],
"aDAI": [
{
"query": "./mainnet.graphql",
"variables": {
"address": "0x028171bca77440897b824ca71d1c56cac55b68a3",
"network": "MATIC"
}
}
]
}
}
},
"menus": {
"aave-all": [
"mainnet.aave",
"polygon.aave"
]
}
}$> w3 query -f ./cookbook.json 'mainnet'
$> w3 query -f ./cookbook.json 'mainnet polygon.aave.aUSDC'
$> w3 query -f ./cookbook.json 'mainnet.aave'
$> w3 query -f ./cookbook.json 'mainnet.aave.uUSDC'
$> w3 query -f ./cookbook.json 'aave-all'
$> w3 query -f ./cookbook.json 'aave-all polygon.aave.aDAI'
cli/src/commands/query.tsapi fieldrecipes objectmenus object into the recipes object, erroring out on any name conflictsrecipes object:
Here's an example of what I figure the implementation should probably look like (note, this is just a spike and is by no means well-tested or production-ready or anything like that; it's just something to give inspiration for the actual implementation):
interface Query {
query: string;
variables: {
[variable: string]: any
};
}
async function _executeQuery(api: string, q: Query): Promise<void> {
// execute the query here
}
function _parse(query: string): string[][] {
return query.split(' ').map(q => q.split('.'));
}
function _resolve(cookbook: object, query: string[]): Query[] {
const val = query.reduce((acc, cur) => acc?.[cur], cookbook);
if (val == null) throw `Failed to resolve query: could not find "${query.join('.')}" in the cookbook`;
if (Array.isArray(val)) {
if (typeof val[0] === 'string') return val.flatMap(menu => _resolve(cookbook, _parse(menu)[0]));
else return val as Query[];
} else return Object.keys(val).flatMap(k => _resolve(val[k], [k]));
}
function query(cookbookFile: string, query: string): void {
const {api, recipes, menus} = JSON.parse(cookbookFile);
if (Object.keys(menus).some(k => k in recipes)) throw 'Name collision! One of your menus has the same name as a recipe';
Object.assign(recipes, menus);
Object.freeze(recipes);
_parse(query)
.map(q => _resolve(recipes, q))
.flat()
.forEach(q => _executeQuery(api, q));
}-f flag and namespacing the recipes by the name of the cookbook file (e.g. cookbook.aave-all)As it has been presented thus far (in this comment and the ones above), this is a good first step: JSON is an incredibly popular format with de-/serializers available in most languages, so it arguably has the lowest cost of use of any reasonable data format; the described ontology is easily extensible to include other QoL features that devs might expect, and can be made more ergonomic incrementally; in short, it gets the job done. However, given the future aspirations of Polywrap (e.g. visual coding/GUI), I'm of the opinion that a more graph-theoretic or category-theoretic (or both) approach should be taken to the general problem of query recipes, and that JSON will become more of a hindrance than a boon. I've compiled a bit of light reading below that may possibly come in handy at such a future time, just to get the ideas flowing about how to approach query recipes (e.g. I think the idea of algebraic effects is a great feature to consider when making a pipeline-like system).
evanjacobs
commented
2 years ago I like the flexibility of the tags-based approach but there seems to be some redundancy (e.g. defining "network" in both "tags" and "variables") which could also introduce unexpected behavior if those values disagree.
Niraj-Kamdar
commented
2 years ago @pwvpwvpwv We should also add support for writing recipes in yaml instead of json to support comments and be less verbose. You should also checkout this issue #648 which goes hand in hand with this one:
dOrgJelli
commented
2 years ago This should be completed by this PR: https://github.com/polywrap/monorepo/pull/903
Currently
npx w3 query e2e.jsonrun every recipes in thee2e.jsonfile but it'd nice if we can specify a particular recipe by name or group and run itHere are different ways which we can use to implement this feature
1) add two new fields for each recipes:
groupandname.Example e2e file would look like:
Command may look like following:
Where
recipe_groupwould be analogous to jest's describe function andrecipe_namewould be analogous to jest's it function.npx w3 query e2e.json -t "Mainnet"would only run task with group Mainnet.npx w3 query e2e.json -t "Mainnet aUSDC"would only run task with group=Mainnet and name=USDC.2) allow user to define custom tags for the recipes:
Example e2e file would look like:
Command may look like following:
This would allow user to define more specific tags for the recipe. Although command line validation and parsing of tags can costlier for large file and/or more tags.
3) Change the recipes schema into recursive style to increase specificity
Example e2e file would look like:
Command may look like following:
This can be n-dimensional theoretically and it would even be easier to run all commands due to hierarchy being preserved as key of the object. A con is recursive nature of the schema.