Add compute resource usage to processing.json

[camilolaiton]

It is important to track compute resources whenever we are running our scripts in order to optimize our algorithms. Therefore, I propose that we include that in the processing.json. @dyf @saskiad

These are some of the important info I am adding to the new versions of the SmartSPIM capsules:

Information about operative system, compute resources and its peak usage could be very important but could not be limited to only these. The usage of compute resources over time/step could also be very important to track.

[saskiad]

I think we should move on this - the idea has come up in a couple of conversations and I think implementing this would be helpful.

[saskiad]

need specs

[dbirman]

@camilolaiton is going to provide some more details on what we might want to include here, copying some possible things from above to discuss:

• OS
• CPU info
• GPU info
• cpu/gpu/ram usage snapshots over time
• instance information (cpu/gpu/ram/memory)
• instance costs? not sure how visible that is

[camilolaiton]

Personally, I'd like a schema that includes:

1. Operative System (e.g., Linux, Windows, Mint, MacOS, ...) and its version.
2. Machine architecture (e.g., x86_64, we should definitely include bit version).
3. Physical resources (e.g., number of CPU cores, RAM, shared memory compartment, number of GPUs, Disks).
4. Usage of CPU as a percentage over time (I don't think it should be that specific as % per core), same for GPUs and RAM. Maybe include SHM as well if you guys agree.

I think those are the most important ones for reproducibility and track our algorithms and used resources. I like @dbirman idea of instance costs but that'd work only if we register the type of machine that was used. I see it more as another script that based on this info could compute the money we spent.

[saskiad]

I think we should hold off on cost in the schema itself. We're making tools to track that - we definitely want eyes on it - but I have reservations about putting it directly into the schema. any thoughts @dyf ?

[dbirman]

I'll make a draft with the above, minus cost, and then we can iterate from there.

[dbirman]

(blocked waiting on https://github.com/AllenNeuralDynamics/aind-data-schema-models/pull/75)

[dbirman]

• [x] Operating system enum
• [x] Architecture enum
• [x] ResourceTimestamp refactor (TimestampedValue in models?)

[dbirman]

Waiting on https://github.com/AllenNeuralDynamics/aind-data-schema-models/issues/77