Need better granularity in glideclient_manifests /glideclientglobal_manifests publisher to be able to publish classads for some VO's / groups and not others

[StevenCTimm]

Currently the logic engine in gpde01 channel gp_resource_request is configured to do the following:

  "logicengines": {
    "logicengine1": {
      "module": "decisionengine.framework.logicengine.LogicEngine",
      "name": "LogicEngine",
      "parameters": {
        "rules": {
          "publish_glidein_requests": {
            "expression": "(publish_requests)",
            "actions": [
              "glideclientglobal_manifests",
              "glideclient_manifests"
            ],
            "facts": []
          },
          "publish_grid_requests": {
            "expression": "(allow_grid)",
            "actions": [],
            "facts": [
              "allow_grid_requests"
            ]
          },
          "publish_aws_requests": {
            "expression": "(awswithininstburnrate && awswithinbillburnrate && aw
sabovebalance)",
            "actions": [],
            "facts": [
              "allow_aws_requests"
            ]
          },
          "publish_gce_requests": {
            "expression": "(gcewithininstburnrate && gceabovebalance)",
            "actions": [],
            "facts": [
              "allow_gce_requests"
            ]
          },
          "publish_lcf_requests": {
            "expression": "(fifenerscbelowlimit)",
            "actions": [],
            "facts": [
              "allow_lcf_requests"
            ]
          }
        },
        "facts": {
          "publish_requests": "(True)",
          "allow_grid": "(True)",
          "allow_lcf": "(True)",
          "allow_gce": "(True)",
          "allow_aws": "(True)",
          "awswithininstburnrate": "financial_params.iloc[0].target_aws_vm_burn_
rate>AWS_Burn_Rate.iloc[0].BurnRate",
          "awswithinbillburnrate": "financial_params.iloc[0].target_aws_bill_bur
n_rate>AWS_Billing_Rate[AWS_Billing_Rate['accountName']=='Fermilab'].iloc[0].cos
tRatePerHourInLastSixHours",
          "awsabovebalance": "financial_params.iloc[0].target_aws_balance<AWS_Bi
lling_Info[AWS_Billing_Info['AccountName']=='Fermilab'].iloc[0].Balance",
          "gcewithininstburnrate": "financial_params.iloc[0].target_gce_vm_burn_
rate>GCE_Burn_Rate.iloc[0].BurnRate",
          "gceabovebalance": "financial_params.iloc[0].target_gce_balance<GCE_Bi
lling_Info.iloc[0].Balance",
          "fifenerscbelowlimit": "Nersc_Allocation_Info[Nersc_Allocation_Info['n
ame']=='fife'].iloc[0].usedAlloc<Nersc_Allocation_Info[Nersc_Allocation_Info['na
me']=='fife'].iloc[0].currentAlloc"
        }
      }
    }
  },

The publishers glideclient_manifests /glideclientglobal_manifests are called based on the expression _publishrequests which works out to be always true.

There are four other Facts that are set and stored in a data block by the Logic Engine, namely allow_grid_requests, allow_aws_requests, allow_gce_requests, and allow_lcf_requests.

The Publisher is called to Consume two datablocks, the glideclientglobal_manifests and the glideclient_manifests, both of which already include fully formed classads containing all resource types

If the allow_lcf_requests flag is true then all requests for LCF (which in current configuration is only NERSC) are published unconditionally. If false, then they are all dropped unconditionally.

This whole section of resource requests and the transform that makes the classads are supposed to be totally rewritten anyway, but when they are, then we need to have the capacity to selectively publish some NERSC request classads and not others, based on which VO is authorized to use NERSC at any given time and has allocation to do so. For instance we should be able to publish classads for GM2 and not for DUNE if GM2 still has allocation and DUNE does not.

This could be accomplished on a group by group basis for instance, if we continue having the analog of frontend groups such as dune_nersc_passthrough, gm2_nersc_passthrough, nova_nersc_passthrough. You could send only those glideclient classads for those that have the quota as determined by the logic engine.

[mambelli]

@StevenCTimm This changed w/ provides/consumes decorators. I think it has been simplified but the need for the change is still there. Am I correct? Is the syntax different now?

[StevenCTimm]

No this has nothing to do with provides/consumes decorators. This has to do with the flags that are set in the Decision engine above and passed to the publishers. Right now there is only one binary flag, either publish all or none of the AWS classads, all or none of the GCE classads, all or none of the LCF classads. We need to be able to control that on a VO by VO basis.

[StevenCTimm]

This issue now urgent because for the first time we have two different FIFE VO's running on NERSC simultaneously, would like to make sure in the channel redesign this feature takes priority.