Need to sanitize model output with postprocessing function to ensure it is a numpy array (assuming this is the expected type). Additionally need some more verbose type information 😅
Steps To Reproduce
Run MITM objdet with export_batches on
Additional Information
$ python -m armory run ./eval7/mitm_eval7_objdet/advpatch_rgb.json --num-eval-batches=1
2023-08-17 17:51:58 0s INFO armory.eval.evaluator:init:30 Constructing Evaluator Object
2023-08-17 17:51:58 0s INFO armory.eval.evaluator:init:61 Retrieving Environment Variables
2023-08-17 17:51:58 0s SUCCESS armory.docker.images:is_image_local:133 found docker image mitm_eval7_objdet:1.1
2023-08-17 17:51:59 2s INFO armory.docker.management:init:62 ARMORY Instance 7d8b44bd5d1b created.
2023-08-17 17:51:59 2s INFO armory.eval.evaluator:_run_config:287 Running evaluation script
2023-08-17 17:52:00 0s INFO armory.delayed_imports::7 Importing and configuring torch, tensorflow, and art, if available. This may take some time.
2023-08-17 17:52:04 4s INFO armory.delayed_imports::30 Setting tf.config.experimental.set_memory_growth to True on all GPUs
2023-08-17 17:52:04 4s INFO art.config:set_data_path:55 set ART_DATA_PATH to /tmp/.art/data
2023-08-17 17:52:05 5s INFO art.config:set_data_path:55 set ART_DATA_PATH to /armory/saved_models/art
2023-08-17 17:52:07 7s INFO matplotlib.font_manager:_load_fontmanager:1624 generated new fontManager
2023-08-17 17:52:11 10s INFO armory.data.utils:maybe_download_weights_from_s3:47 Using available MITM-objdet_rgb.pt in Armory saved_model_dir
2023-08-17 17:52:16 16s INFO armory.scenarios.scenario:load_dataset:257 Loading test dataset carla_over_obj_det_dev...
2023-08-17 17:52:16 16s WARNING tensorflow.python.platform.tf_logging:warning:173 From /opt/conda/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23.
Instructions for updating:
Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. https://github.com/tensorflow/tensorflow/issues/56089
2023-08-17 17:52:16 16s INFO armory.scenarios.scenario:evaluate_all:335 Running inference on benign and adversarial examples
Evaluation: 0% 0/1 [00:00<?, ?it/s]
Evaluation: 0% 0/1 [00:06<?, ?it/s]
2023-08-17 17:52:22 22s ERROR armory.scenarios.scenario:evaluate:452 Encountered error during scenario evaluation.
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
│ │ └ {'name': 'main', 'doc': '\nMain script for running scenarios. Users will run a scenario by calling:\n armory r...
│ └ <code object at 0x7fe22f6bb240, file "/workspace/armory/scenarios/main.py", line 1>
â”” <function _run_code at 0x7fe31a85f9d0>
File "/opt/conda/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
│ └ {'name': 'main', 'doc': '\nMain script for running scenarios. Users will run a scenario by calling:\n armory r...
â”” <code object at 0x7fe22f6bb240, file "/workspace/armory/scenarios/main.py", line 1>
File "/workspace/armory/scenarios/main.py", line 229, in
run_config(
â”” <function run_config at 0x7fe22e52e1f0>
File "/workspace/armory/scenarios/main.py", line 145, in run_config
scenario.evaluate()
│ └ <function Scenario.evaluate at 0x7fe202cdf670>
â”” <armory.scenarios.carla_object_detection.CarlaObjectDetectionTask object at 0x7fe22f385730>
File "/workspace/armory/scenarios/scenario.py", line 443, in evaluate
self._evaluate()
│ └ <function Scenario._evaluate at 0x7fe202cdf5e0>
â”” <armory.scenarios.carla_object_detection.CarlaObjectDetectionTask object at 0x7fe22f385730>
File "/workspace/armory/scenarios/scenario.py", line 433, in _evaluate
self.evaluate_all()
│ └ <function Scenario.evaluate_all at 0x7fe202cdf1f0>
â”” <armory.scenarios.carla_object_detection.CarlaObjectDetectionTask object at 0x7fe22f385730>
File "/workspace/armory/scenarios/scenario.py", line 338, in evaluate_all
self.evaluate_current()
│ └ <function Scenario.evaluate_current at 0x7fe202cdf4c0>
â”” <armory.scenarios.carla_object_detection.CarlaObjectDetectionTask object at 0x7fe22f385730>
File "/workspace/armory/scenarios/scenario.py", line 417, in evaluate_current
self.run_benign()
│ └ <function Scenario.run_benign at 0x7fe202cdf3a0>
â”” <armory.scenarios.carla_object_detection.CarlaObjectDetectionTask object at 0x7fe22f385730>
File "/workspace/armory/scenarios/scenario.py", line 363, in run_benign
self.probe.update(y_pred=y_pred)
│ │ │ └ [{'boxes': tensor([[ 417.6945, 667.3348, 547.8636, 767.3787],
│ │ │ [ 446.4780, 769.8671, 609.9299, 900.4496],
│ │ │ ...
│ │ └ <function Probe.update at 0x7fe202e2b700>
│ └ <armory.instrument.instrument.Probe object at 0x7fe22e4ddf40>
â”” <armory.scenarios.carla_object_detection.CarlaObjectDetectionTask object at 0x7
fe22f385730>
File "/workspace/armory/instrument/instrument.py", line 118, in update
self.sink.update(name, value)
│ │ │ │ └ [{'boxes': tensor([[ 417.6945, 667.3348, 547.8636, 767.3787],
│ │ │ │ [ 446.4780, 769.8671, 609.9299, 900.4496],
│ │ │ │ ...
│ │ │ └ 'scenario.y_pred'
│ │ └ <function Hub.update at 0x7fe202e3d0d0>
│ └ <armory.instrument.instrument.Hub object at 0x7fe22e4dd9a0>
â”” <armory.instrument.instrument.Probe object at 0x7fe22e4ddf40>
File "/workspace/armory/instrument/instrument.py", line 305, in update
meter.set(arg, value, self.context["batch"])
│ │ │ │ │ └ {'batch': 0, 'stage': 'benign'}
│ │ │ │ └ <armory.instrument.instrument.Hub object at 0x7fe22e4dd9a0>
│ │ │ └ [{'boxes': tensor([[ 417.6945, 667.3348, 547.8636, 767.3787],
│ │ │ [ 446.4780, 769.8671, 609.9299, 900.4496],
│ │ │ ...
│ │ └ 'scenario.y_pred'
│ └ <function Meter.set at 0x7fe202e3d670>
â”” <armory.instrument.export.ExportMeter object at 0x7fe1e9ecfeb0>
File "/workspace/armory/instrument/instrument.py", line 507, in set
self.measure()
│ └ <function ExportMeter.measure at 0x7fe202cc7ca0>
â”” <armory.instrument.export.ExportMeter object at 0x7fe1e9ecfeb0>
File "/workspace/armory/instrument/export.py", line 682, in measure
self.exporter.export(
│ │ └ <function SampleExporter.export at 0x7fe202cb8e50>
│ └ <armory.instrument.export.ObjectDetectionExporter object at 0x7fe1e9ecfc10>
â”” <armory.instrument.export.ExportMeter object at 0x7fe1e9ecfeb0>
File "/workspace/armory/instrument/export.py", line 31, in export
self._export(
│ └ <function ObjectDetectionExporter._export at 0x7fe202cc7310>
â”” <armory.instrument.export.ObjectDetectionExporter object at 0x7fe1e9ecfc10>
File "/workspace/armory/instrument/export.py", line 125, in _export
self.image_with_boxes = self.get_sample(
│ │ └ <function ObjectDetectionExporter.get_sample at 0x7fe202cc73a0>
│ └ <armory.instrument.export.ObjectDetectionExporter object at 0x7fe1e9ecfc10>
â”” <armory.instrument.export.ObjectDetectionExporter object at 0x7fe1e9ecfc10>
File "/workspace/armory/instrument/export.py", line 181, in get_sample
pred_box.astype("float32"), outline="white", width=
2
â”” tensor([417.6945, 667.3348, 547.8636, 767.3787])
AttributeError: 'Tensor' object has no attribute 'astype'
2023-08-17 17:52:25 27s ERROR armory.docker.management:exec_cmd:105 command python -m armory.scenarios.main eyJfZGVzY3JpcHRpb24iOiAiQ0FSTEEgc2luZ2xlIG1vZGFsaXR5IG9iamVjdCBkZXRlY3Rpb24sIGNvbnRyaWJ1dGVkIGJ5IE1JVFJFIENvcnBvcmF0aW9uIiwgImFkaG9jIjogbnVsbCwgImF0dGFjayI6IHsia25vd2xlZGdlIjogIndoaXRlIiwgImt3YXJncyI6IHsiYmF0Y2hfc2l6ZSI6IDQsICJsZWFybmluZ19yYXRlIjogMC4wMDMsICJtYXhfaXRlciI6IDEsICJvcHRpbWl6ZXIiOiAicGdkIiwgInRhcmdldGVkIjogZmFsc2UsICJ2ZXJib3NlIjogdHJ1ZX0sICJtb2R1bGUiOiAiYXJtb3J5LmFydF9leHBlcmltZW50YWwuYXR0YWNrcy5jYXJsYV9vYmpfZGV0X2FkdmVyc2FyaWFsX3BhdGNoIiwgIm5hbWUiOiAiQ0FSTEFBZHZlcnNhcmlhbFBhdGNoUHlUb3JjaCIsICJ1c2VfbGFiZWwiOiB0cnVlfSwgImRhdGFzZXQiOiB7ImJhdGNoX3NpemUiOiAxLCAiZXZhbF9zcGxpdCI6ICJkZXYiLCAiZnJhbWV3b3JrIjogIm51bXB5IiwgIm1vZGFsaXR5IjogInJnYiIsICJtb2R1bGUiOiAiYXJtb3J5LmRhdGEuYWR2ZXJzYXJpYWxfZGF0YXNldHMiLCAibmFtZSI6ICJjYXJsYV9vdmVyX29ial9kZXRfZGV2In0sICJkZWZlbnNlIjogbnVsbCwgIm1ldHJpYyI6IHsibWVhbnMiOiB0cnVlLCAicGVydHVyYmF0aW9uIjogImwwIiwgInJlY29yZF9tZXRyaWNfcGVyX3NhbXBsZSI6IGZhbHNlLCAidGFzayI6IFsiY2FybGFfb2RfQVBfcGVyX2NsYXNzIiwgImNhcmxhX29kX2Rpc2FwcGVhcmFuY2VfcmF0ZSIsICJjYXJsYV9vZF9oYWxsdWNpbmF0aW9uc19wZXJfaW1hZ2UiLCAiY2FybGFfb2RfbWlzY2xhc3NpZmljYXRpb25fcmF0ZSIsICJjYXJsYV9vZF90cnVlX3Bvc2l0aXZlX3JhdGUiLCAib2JqZWN0X2RldGVjdGlvbl9tQVBfdGlkZSJdfSwgIm1vZGVsIjogeyJmaXQiOiBmYWxzZSwgImZpdF9rd2FyZ3MiOiB7fSwgIm1vZGVsX2t3YXJncyI6IHsibnVtX2NsYXNzZXMiOiAzLCAibW9kYWxpdHkiOiAicmdiIn0sICJtb2R1bGUiOiAicm9idXN0X29iamRldCIsICJuYW1lIjogImdldF9hcnRfbW9kZWwiLCAid2VpZ2h0c19maWxlIjogIk1JVE0tb2JqZGV0X3JnYi5wdCIsICJ3cmFwcGVyX2t3YXJncyI6IHt9fSwgInNjZW5hcmlvIjogeyJrd2FyZ3MiOiB7fSwgImV4cG9ydF9iYXRjaGVzIjogdHJ1ZSwgIm1vZHVsZSI6ICJhcm1vcnkuc2NlbmFyaW9zLmNhcmxhX29iamVjdF9kZXRlY3Rpb24iLCAibmFtZSI6ICJDYXJsYU9iamVjdERldGVjdGlvblRhc2sifSwgInN5c2NvbmZpZyI6IHsiZG9ja2VyX2ltYWdlIjogIm1pdG1fZXZhbDdfb2JqZGV0OjEuMSIsICJleHRlcm5hbF9naXRodWJfcmVwbyI6IG51bGwsICJsb2NhbF9yZXBvX3BhdGgiOiAibWl0bV9ldmFsN19vYmpkZXQiLCAiZ3B1cyI6ICJhbGwiLCAib3V0cHV0X2RpciI6IG51bGwsICJvdXRwdXRfZmlsZW5hbWUiOiBudWxsLCAidXNlX2dwdSI6IHRydWUsICJmaWxlcGF0aCI6ICIuL2V2YWw3L21pdG1fZXZhbDdfb2JqZGV0L2FkdnBhdGNoX3JnYi5qc29uIiwgIm51bV9ldmFsX2JhdGNoZXMiOiAxfSwgImV2YWxfaWQiOiAiMjAyMy0wOC0xN1QxNzUxNTguMDk1OTIwIn0= --num-eval-batches 1 --base64 did not finish cleanly
2023-08-17 17:52:25 27s INFO armory.docker.management:stop_armory_instance:141 Stopping instance: 7d8b44bd5d1b
2023-08-17 17:52:25 27s INFO armory.eval.evaluator:_cleanup:134 deleting tmp_dir /nfs/gard/aruckman/git/twosixlabs/.armory/tmp/2023-08-17T175158.095920
2023-08-17 17:52:25 27s INFO armory.eval.evaluator:_cleanup:153 results output written to:
/nfs/gard/aruckman/git/twosixlabs/.armory/outputs/2023-08-17T175158.095920/
Description of the bug
Need to sanitize model output with postprocessing function to ensure it is a numpy array (assuming this is the expected type). Additionally need some more verbose type information 😅
Steps To Reproduce
Run MITM objdet with
export_batchesonAdditional Information
$ python -m armory run ./eval7/mitm_eval7_objdet/advpatch_rgb.json --num-eval-batches=1 2023-08-17 17:51:58 0s INFO armory.eval.evaluator:init:30 Constructing Evaluator Object 2023-08-17 17:51:58 0s INFO armory.eval.evaluator:init:61 Retrieving Environment Variables 2023-08-17 17:51:58 0s SUCCESS armory.docker.images:is_image_local:133 found docker image mitm_eval7_objdet:1.1 2023-08-17 17:51:59 2s INFO armory.docker.management:init:62 ARMORY Instance 7d8b44bd5d1b created. 2023-08-17 17:51:59 2s INFO armory.eval.evaluator:_run_config:287 Running evaluation script 2023-08-17 17:52:00 0s INFO armory.delayed_imports::7 Importing and configuring torch, tensorflow, and art, if available. This may take some time.
2023-08-17 17:52:04 4s INFO armory.delayed_imports::30 Setting tf.config.experimental.set_memory_growth to True on all GPUs
2023-08-17 17:52:04 4s INFO art.config:set_data_path:55 set ART_DATA_PATH to /tmp/.art/data
2023-08-17 17:52:05 5s INFO art.config:set_data_path:55 set ART_DATA_PATH to /armory/saved_models/art
2023-08-17 17:52:07 7s INFO matplotlib.font_manager:_load_fontmanager:1624 generated new fontManager
2023-08-17 17:52:11 10s INFO armory.data.utils:maybe_download_weights_from_s3:47 Using available MITM-objdet_rgb.pt in Armory at 0x7fe22f6bb240, file "/workspace/armory/scenarios/main.py", line 1>
â”” <function _run_code at 0x7fe31a85f9d0>
File "/opt/conda/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
│ └ {'name': 'main', 'doc': '\nMain script for running scenarios. Users will run a scenario by calling:\n armory r...
â”” <code object at 0x7fe22f6bb240, file "/workspace/armory/scenarios/main.py", line 1>
File "/workspace/armory/scenarios/main.py", line 229, in
run_config(
â”” <function run_config at 0x7fe22e52e1f0>
File "/workspace/armory/scenarios/main.py", line 145, in run_config
scenario.evaluate()
│ └ <function Scenario.evaluate at 0x7fe202cdf670>
â”” <armory.scenarios.carla_object_detection.CarlaObjectDetectionTask object at 0x7fe22f385730>
saved_model_dir2023-08-17 17:52:16 16s INFO armory.scenarios.scenario:load_dataset:257 Loading test dataset carla_over_obj_det_dev... 2023-08-17 17:52:16 16s WARNING tensorflow.python.platform.tf_logging:warning:173 From /opt/conda/lib/python3.9/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23. Instructions for updating: Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. https://github.com/tensorflow/tensorflow/issues/56089 2023-08-17 17:52:16 16s INFO armory.scenarios.scenario:evaluate_all:335 Running inference on benign and adversarial examples Evaluation: 0% 0/1 [00:00<?, ?it/s] Evaluation: 0% 0/1 [00:06<?, ?it/s] 2023-08-17 17:52:22 22s ERROR armory.scenarios.scenario:evaluate:452 Encountered error during scenario evaluation. Traceback (most recent call last): File "/opt/conda/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, │ │ └ {'name': 'main', 'doc': '\nMain script for running scenarios. Users will run a scenario by calling:\n armory r... │ └ <code object