feature request: programmatic interface for running optimisations

[chriswessels]

Is your feature request related to a problem? Please describe.

When running optimisations, I want a structured interface to pass data into the optimiser (indexer prefs) and a structured interface to get optimisation results out

Describe the solution you'd like

Few options:

1. A flag that allows passing indexer preferences as JSON object via stdin, and a flag that allows providing optimisation outcomes as a JSON object via stdout (e.g. --output json)
2. A structured request/response API (e.g. GraphQL) to allow performing optimisations via an API call

Describe alternatives you've considered

Spawning the process and using sed and other tools to extract some output

[aasseman]

@chriswessels you can relatively easily wrap Julia into Python. Here's something I'm currently tinkering with:

from decimal import Decimal, getcontext
from typing import Dict, Mapping, Sequence

from juliacall import Main as jl
from juliacall import convert

from allocopt_wrapper import config

# set Decimal precision to 18 places
getcontext().prec = 18

def optimized_allocations(
    whitelist: Sequence[str],
    blacklist: Sequence[str],
    pinnedlist: Sequence[str],
    frozenlist: Sequence[str],
) -> Dict[str, Decimal]:
    """Generate optimized allocations using the settings in `config`.

    Args:
        whitelist (Sequence[str]): List of subgraph IPFS hashes to whitelist.
        blacklist (Sequence[str]): List of subgraph IPFS hashes to blacklist.
        pinnedlist (Sequence[str]): List of subgraph IPFS hashes to pin.
        frozenlist (Sequence[str]): List of subgraph IPFS hashes to freeze.

    Returns:
        Dict[str, float]: Dictionnary of subgraph deployments and amount to allocate
            in GRT.
    """
    jl.seval("using AllocationOpt")

    whitelist = convert(jl.Array[jl.String], whitelist)
    blacklist = convert(jl.Array[jl.String], blacklist)
    pinnedlist = convert(jl.Array[jl.String], pinnedlist)
    frozenlist = convert(jl.Array[jl.String], frozenlist)
    grtgas = jl.Float64(config.MAX_GRT_GAS_PER_ALLOCATION)
    allocation_lifetime = jl.Int64(config.ALLOCATION_LIFETIME)

    repo, indexer, network = jl.network_state(
        config.INDEXER_ADDRESS,
        1,
        whitelist,
        blacklist,
        pinnedlist,
        frozenlist,
        config.THEGRAPH_NETWORK_SUBGRAPH_ENDPOINT,
    )
    fullrepo, _, _ = jl.network_state(
        config.INDEXER_ADDRESS,
        1,
        jl.Array[jl.String]([]),
        jl.Array[jl.String]([]),
        jl.Array[jl.String]([]),
        jl.Array[jl.String]([]),
        config.THEGRAPH_NETWORK_SUBGRAPH_ENDPOINT,
    )

    filter_fn = jl.seval(
        "(network, grtgas, allocation_lifetime)"
        " -> (ω, ψ, Ω)"
        " -> apply_preferences(network, grtgas, allocation_lifetime, ω, ψ, Ω)"
    )(network, grtgas, allocation_lifetime)

    omega: Mapping[str, float] = jl.optimize_indexer(
        indexer,
        repo,
        fullrepo,
        config.MAX_NEW_ALLOCATIONS,
        config.NONGREEDINESS,
        filter_fn,
        pinnedlist,
    )

    return {k: Decimal(v) for k, v in omega.items()}

PS: Fixed conversion error from python lists to julia arrays, converting output GRT amounts to Decimal.

[hopeyen]

There hasn't been any response in a while so we will close for now. This is not planned in the optimizer scope in addition of the ^ wrapper