A script to check if all the files / globs in the sources of a dataset configuration (YAML) file are valid. (The script ensures that files are properly getting referenced in the config file, so that the process of creating dataset configs is easier)
Currently only supports run_data_map_file, but looking to extend to other standardization_config fields in the future.
Requires AWS credentials to access the S3 bucket where the files are stored.
Attempts to replace any formatted strings with their corresponding values from the run_data_map_file.
A script to check if all the files / globs in the sources of a dataset configuration (YAML) file are valid. (The script ensures that files are properly getting referenced in the config file, so that the process of creating dataset configs is easier) Currently only supports run_data_map_file, but looking to extend to other standardization_config fields in the future. Requires AWS credentials to access the S3 bucket where the files are stored. Attempts to replace any formatted strings with their corresponding values from the run_data_map_file.