search

nrrb / noaa-gsod-data-munging

Munging the NOAA GSOD data into CSV and SQL.
MIT License
13 stars 4 forks source link

Argument list too long #1

Open fhoffa opened 9 years ago

fhoffa commented 9 years ago

In https://github.com/tothebeat/noaa-gsod-data-munging/blob/master/get_data_for_year_XXXX.sh#L19:

 for filename in `ls *.op`; do   gunzip $filename; done

gives "Argument list too long"

Fix:

 find . -name "*.gz" -print0 | xargs -0 gunzip
fhoffa commented 9 years ago

For the concatenation:

find . -name "*.op" -print0 | xargs -0 grep -h -v STN  > ../2014.op
fhoffa commented 9 years ago

Left my (working) script at http://www.reddit.com/r/bigquery/comments/2ts9wo/noaa_gsod_weather_data_loaded_into_bigquery/

nrrb commented 9 years ago

Wow, nice work @fhoffa! Thanks for providing a fix! I didn't think anyone else ever looked at this besides me.

This might prompt me into looking at BigQuery this weekend, revisiting this weather data, and correcting my script.