usnistgov / dioptra

Test Software for the Characterization of AI Technologies
https://pages.nist.gov/dioptra/
Other
211 stars 30 forks source link

[BUG] RESTAPI seems to convert null parameter values to a string 'None' #631

Open jtsextonMITRE opened 3 weeks ago

jtsextonMITRE commented 3 weeks ago

I believe this is a separate bug from #607

While #607 deals with default parameters in the parameters: section of the entrypoint YAML, this is for default literal values of inputs within the task graph. From what I can tell, at least some of these inputs are processed properly, as integers do seem to show up as integers.

For example, this correctly processes .01 and 5 as a float and an integer respectively:

  callbacks:
    get_callbacks:
      - - name: EarlyStopping
          parameters:
            monitor: val_loss
            min_delta: .01
            patience: 5
            restore_best_weights: true
    dependencies:
      - init_tensorflow

However, when I read this in from the YAML file on the client side via python, the null in YAML becomes None in python, which when uploaded to the RESTAPI becomes "None":

  testing_dataset:
    create_dataset:
      data_dir: $testing_dir
      subset: null
      image_size: $image_size
      seed: $dataset_seed
      validation_split: null
      batch_size: $batch_size
    dependencies:
      - init_tensorflow

In this example, subset does not throw a validation error because it is typed as union: [string, "null"] (so string is a valid type), but validation_split is typed as union: [number, "null"], so the validation fails, but both are ultimately wrong as neither are None by the time it reaches the task engine.

I also tried with directly uploading a string "null", which just stays as a string "null" all the way through the task engine.

Long story short if we don't have one already I think there needs to be a way to specify nulls through strings, which seems to have been lost since we're no longer directly uploading the YAML file from the client to the worker.

jtsextonMITRE commented 3 weeks ago

after more testing, I believe this was just the fault of uploading JSON-formatted yaml instead of uploading portions of the yaml file directly for the task graph.