The skimmer currently reads the parameter.txt file which stores info like
### Dates to process
year_start 2016
year_end 2017
month_start 1
month_end 12
day_start 1
day_end 31
These variables are read out in:
-rwxr-x--- 1 dampe_prod dampe 4208 Dec 11 13:50 check_files.bash
-rwxr-x--- 1 dampe_prod dampe 2461 Dec 4 13:58 check_skimmed_files.bash
-rwxr-x--- 1 dampe_prod dampe 4048 Dec 11 13:50 get_file_lists.bash
-rwxr-x--- 1 dampe_prod dampe 6769 Nov 30 14:49 get_summary.bash
-rwxr-x--- 1 dampe_prod dampe 3907 Dec 4 14:00 make_summary_plot.bash
with a preamble that looks like this
year_start="`cat ../../parameters.txt | grep year_start | awk '{print $2}'`"
year_end="`cat ../../parameters.txt | grep year_end | awk '{print $2}'`"
month_start="`cat ../../parameters.txt | grep month_start | awk '{print $2}'`"
month_end="`cat ../../parameters.txt | grep month_end | awk '{print $2}'`"
day_start="`cat ../../parameters.txt | grep day_start | awk '{print $2}'`"
day_end="`cat ../../parameters.txt | grep day_end | awk '{print $2}'`"
[...]
for year in $(seq $year_start $year_end)
do
for month in $(seq $month_start $month_end)
do
month=`printf "%02d" ${month}`
dir="${output_location}/${year}/${month}"
if [ ! -d ${dir} ]; then continue; fi
cd ${dir} > /dev/null
for day in $(seq $day_start $day_end)
[...]
However, in this way, one can only skim one year at a time. So for instance, if one needed/wanted to re-skim with a more recent release, one would need to skim at least 2x, once for the completed years and once for the started year. This should be reviewed. A potential way to simplify things could be to replace the above block:
this way, if the variables ${skimmer_year_start} and ${skimmer_year_end} exist they will be taken, otherwise the parameter file will be read out. In this way, a cron job could simply iterate through these variables and skimmer jobs can be dispatched on an automatic basis.
The skimmer currently reads the parameter.txt file which stores info like
These variables are read out in:
with a preamble that looks like this
However, in this way, one can only skim one year at a time. So for instance, if one needed/wanted to re-skim with a more recent release, one would need to skim at least 2x, once for the completed years and once for the started year. This should be reviewed. A potential way to simplify things could be to replace the above block:
with something akin to:
this way, if the variables ${skimmer_year_start} and ${skimmer_year_end} exist they will be taken, otherwise the parameter file will be read out. In this way, a cron job could simply iterate through these variables and skimmer jobs can be dispatched on an automatic basis.