Open skremdwd opened 1 month ago
I am able to reproduce the issue when running with --map <directory>/
two workarounds (1) use a shell loop to iterate through mapfiles and use --map <filename>.map
(2) --silent
should suppress output.
Issue under investigation...
A normal import process of around 900 files created a total output/log file of around 700 MiB on disk.
It seems like many lines are repeated over and over with the same content, getting another line longer after every imported dataset.
This may be related to #170?
Log output
Part of the output to demonstrate what it looks like, it seems like it is reprinting some parts but I am not sure why it would do that, I can't find a related issue in the publog code.
Part/start of the log file
```text 2024-07-12 08:05:01 INFO No Globus UUID defined. 2024-07-12 08:05:01 INFO No data transfer node defined. 2024-07-12 08:05:01 INFO Converting mapfile... 2024-07-12 08:05:01 INFO Running Extraction... 2024-07-12 08:05:01 INFO Making dataset... 2024-07-12 08:05:01 WARNING experiment does not agree! subseasonal != mfc20240708 2024-07-12 08:05:01 WARNING product does not agree! restricted != output 2024-07-12 08:05:01 INFO Updating... 2024-07-12 08:05:02 INFO [SUCCESS] Number of records updated: 1 2024-07-12 08:05:02 INFO [SUCCESS] Number of records updated: 1 2024-07-12 08:05:02 INFO INFO: Found previous version, updating the record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r77i1p1.DWD-EPISODES2022.v1-r1.day.sfcWind.v20240709|esgf-data.dwd.de 2024-07-12 08:05:02 INFO Running index pub... 2024-07-12 08:05:02 INFO [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r77i1p1.DWD-EPISODES2022.v1-r1.day.sfcWind.v20240709.sfcWind_day_IFS24--DWD-EPISODES2022--HYR-5_mfc20240708_r77i1p1_20240708-20240822.nc|esgf-data.dwd.de 2024-07-12 08:05:02 INFO [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r77i1p1.DWD-EPISODES2022.v1-r1.day.sfcWind.v20240709.sfcWind_day_IFS24--DWD-EPISODES2022--HYR-5_mfc20240708_r77i1p1_20240708-20240822.nc|esgf-data.dwd.de 2024-07-12 08:05:02 INFO [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r77i1p1.DWD-EPISODES2022.v1-r1.day.sfcWind.v20240709|esgf-data.dwd.de 2024-07-12 08:05:02 INFO [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r77i1p1.DWD-EPISODES2022.v1-r1.day.sfcWind.v20240709|esgf-data.dwd.de 2024-07-12 08:05:02 INFO Done. Cleaning up. 2024-07-12 08:05:02 INFO No Globus UUID defined. 2024-07-12 08:05:02 INFO No data transfer node defined. 2024-07-12 08:05:02 INFO Converting mapfile... 2024-07-12 08:05:02 INFO Converting mapfile... 2024-07-12 08:05:02 INFO Running Extraction... 2024-07-12 08:05:02 INFO Running Extraction... 2024-07-12 08:05:02 INFO Making dataset... 2024-07-12 08:05:02 INFO Making dataset... 2024-07-12 08:05:02 WARNING experiment does not agree! subseasonal != mfc20240708 2024-07-12 08:05:02 WARNING experiment does not agree! subseasonal != mfc20240708 2024-07-12 08:05:02 WARNING product does not agree! restricted != output 2024-07-12 08:05:02 WARNING product does not agree! restricted != output 2024-07-12 08:05:02 INFO Updating... 2024-07-12 08:05:02 INFO Updating... 2024-07-12 08:05:02 INFO [SUCCESS] Number of records updated: 1 2024-07-12 08:05:02 INFO [SUCCESS] Number of records updated: 1 2024-07-12 08:05:02 INFO [SUCCESS] Number of records updated: 1 2024-07-12 08:05:02 INFO [SUCCESS] Number of records updated: 1 2024-07-12 08:05:02 INFO [SUCCESS] Number of records updated: 1 2024-07-12 08:05:02 INFO [SUCCESS] Number of records updated: 1 2024-07-12 08:05:02 INFO INFO: Found previous version, updating the record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709|esgf-data.dwd.de 2024-07-12 08:05:02 INFO INFO: Found previous version, updating the record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709|esgf-data.dwd.de 2024-07-12 08:05:02 INFO Running index pub... 2024-07-12 08:05:02 INFO Running index pub... 2024-07-12 08:05:02 INFO [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709.rsds_day_IFS24--DWD-EPISODES2022--HYR-5_mfc20240708_r4i1p1_20240708-20240822.nc|esgf-data.dwd.de 2024-07-12 08:05:02 INFO [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709.rsds_day_IFS24--DWD-EPISODES2022--HYR-5_mfc20240708_r4i1p1_20240708-20240822.nc|esgf-data.dwd.de 2024-07-12 08:05:02 INFO [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709.rsds_day_IFS24--DWD-EPISODES2022--HYR-5_mfc20240708_r4i1p1_20240708-20240822.nc|esgf-data.dwd.de 2024-07-12 08:05:02 INFO [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709.rsds_day_IFS24--DWD-EPISODES2022--HYR-5_mfc20240708_r4i1p1_20240708-20240822.nc|esgf-data.dwd.de 2024-07-12 08:05:02 INFO [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709|esgf-data.dwd.de 2024-07-12 08:05:02 INFO [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709|esgf-data.dwd.de 2024-07-12 08:05:02 INFO [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709|esgf-data.dwd.de 2024-07-12 08:05:02 INFO [SUCCESS] Published record: ClimatePredictionsDE.subseasonal.restricted.output.HYR-5.DWD.IFS24.mvh20240708.mfc20240708.r4i1p1.DWD-EPISODES2022.v1-r1.day.rsds.v20240709|esgf-data.dwd.de 2024-07-12 08:05:02 INFO Done. Cleaning up. 2024-07-12 08:05:02 INFO Done. Cleaning up. 2024-07-12 08:05:02 INFO No Globus UUID defined. 2024-07-12 08:05:02 INFO No data transfer node defined. 2024-07-12 08:05:02 INFO Converting mapfile... 2024-07-12 08:05:02 INFO Converting mapfile... 2024-07-12 08:05:02 INFO Converting mapfile... 2024-07-12 08:05:02 INFO Running Extraction... 2024-07-12 08:05:02 INFO Running Extraction... 2024-07-12 08:05:02 INFO Running Extraction... 2024-07-12 08:05:02 INFO Making dataset... 2024-07-12 08:05:02 INFO Making dataset... 2024-07-12 08:05:02 INFO Making dataset... [...] ```