search

codes-org / codes

The Co-Design of Exascale Storage Architectures (CODES) simulation framework builds upon the ROSS parallel discrete event simulation engine to provide high-performance simulation utilities and models for building scalable distributed systems simulations
Other
40 stars 16 forks source link

CODES SWM Online Workloads: Load app configs at runtime #215

Closed nmcglo closed 3 years ago

nmcglo commented 3 years ago

Currently, when an SWM online workload model is loaded by codes_workload_load() in modelnet-mpi-replay, it reads in the name of the application supplied by the workload configuration file or workload name command line arguments. Deeper down in the implementation of the Online workload loader, it then takes this name that is passed to it, has cascading if elses to find a matching string name for the workload. It then uses that name to find the JSON file that gives information for how this workload should be configured.

The JSON filepaths are hardcoded and are based off of what are copied to the SWM build directory when it's compiled. They can be edited there without needing to recompile but we really should be passing the paths of the workloads in at runtime. Precedence for this is given when we configure DUMPI trace applications: the path to the dumpi trace files given in the workload configuration file.

Proposal: change the way that we load SWM workload configurations so that the JSON filepath is given in the workload conf file that is passed as a command line argument. This JSON file contains a string for the app name, use that app name to know exactly what SWM constructor to call.

Reason: If you want to run many different types of workloads, you have to edit the files in place in the SWM build directory or source tree. This is not friendly for anyone who has over the years run many different workload sets of varying intensity and likes to keep git repos containing all of their simulation configurations and tertiary files for said simulations. I've had to reverse engineer the workload configuration to try and replicate a specific data point in a 5 year old figure and this would have been so much easier if these files were just something the user supplied instead of being in a specific non-user-specified location.

I'll get around to this probably in May/June.

kevinabrown commented 3 years ago

I fully support this idea, @nmcglohon. I would also like us to consider changing the format of the workload config file to use a key-value pair structure. This format could be similar to the model-net config file or we could even use the JSON format. I think this would be useful for new users to quickly understand how to configure workloads and also reduce the mental burden on existing users who currently need to commit the format to memory.

nmcglo commented 3 years ago

Yeah @carothersc and I have talked about de-boostifying SWM at some point if possible. If we did that, we'd have to change this configuration file format to something else (or a different, lighter weight, JSON library). Currently it looks like SWM really only needs the boost header files which makes me think it wouldn't be too hard to de-boost it but time will tell.