error in read.table for picarro

[kaizadp]

@bpbond

https://github.com/PNNL-TES/TES-drydown/blob/740ea83626acfbf47e3902456e3593106f85dbec/code/3d-picarro_output_split.R#L248-L250

Error in read.table(file = file, header = header, sep = sep, quote = quote, : first five rows are empty: giving up

Because the Picarro dataset is sooo big, I've had to extract data for each month and then compile them. I extracted concentration data (ppm), but I'm unable to read and rbind the cpcrw files.

[bpbond]

What does "compile them" mean? It's hard for me to guess what's happening here without a reproducible example. What steps should I take to reproduce this error?

[kaizadp]

I wanted to extract concentration data from the Picarro files, but I couldn’t run script 3c2-picarro_output_ppm.R on the entire dataset because it was so big.

So in script 3d-xx lines 141-241, I processed data one month at a time and saved the ppm data for each month in separate files in “ data/processed/picarro/monthly/monthly_ppm_cpcrw”.

I then try to read all the csv files in that folder and rbind them to make a single file for CPCRW. — and that’s where I get the error. I’ve checked the csv files myself, and I don’t see empty rows.

I’ve done the same thing with SR ppm, without any errors. Same with fluxes too, without errors.

[bpbond]

I'm confused. The 3d script seems to read the same drake target (gf_output), over and over, and write it over and over to differently-named files.

[kaizadp]

I change PICARROPATH each time before re-running the plan, so it’s a new month each run.

[bpbond]

You, my friend, are a monster. 😱

[kaizadp]

Haha, I know 😎. But, shitty laptop. My RStudio session would time out (“memory elapsed”?) when I tried to run the whole thing. Desperate times.

[bpbond]

No, you're not a monster for splitting things up! I get that.

...anyway, I'd really like to try to reproduce this error and help you. Can you give me clear step-by-step instructions?

[kaizadp]

I guess the most direct way would be to just replicate those 3 lines of code.

• download the files in “data/processed/picarro/monthly/monthly_ppm_cpcrw”. Those are the monthly files.
• then run lines 248-251, to read the files and then combine them.

I tried running smaller pieces of that code. The list() command worked and gave me the correct list of files in the folder. But the read.csv doesn’t work.