search

bgctw / RespChamberProc

Processing concentration data from chamber measurements
0 stars 2 forks source link

Regular Timesteps #7

Closed michaelstrupler closed 3 months ago

michaelstrupler commented 3 months ago

The raw timesteps of the PICARRO output look like this, i.e. they are not regular:

DATE TIME FRAC_DAYS_SINCE_JAN1 FRAC_HRS_SINCE_JAN1 JULIAN_DAYS EPOCH_TIME  
<IDat> <char> <num> <num> <num> <num>  
1: 2022-05-31 08:01:59.987 150.2931 3607.033 151.2931 1653976920  
2: 2022-05-31 08:02:01.676 150.2931 3607.034 151.2931 1653976922  
3: 2022-05-31 08:02:03.944 150.2931 3607.034 151.2931 1653976924  
4: 2022-05-31 08:02:04.852 150.2931 3607.035 151.2931 1653976925  
5: 2022-05-31 08:02:06.303 150.2931 3607.035 151.2931 1653976926
TIMESTAMP CO2_dry CH4_dry H2Oppt NH3_dry N2O_dry Collar h AirTemp  
1 2022-05-31 08:01:59 448.4094 1.992880 12.10567 0.1006285 0.2947989 0 153 24  
2 2022-05-31 08:02:00 448.4094 1.992814 12.10567 0.1006285 0.2947989 0 153 24  
3 2022-05-31 08:02:01 448.4094 1.992749 12.10567 0.1006285 0.2947990 0 153 24  
4 2022-05-31 08:02:02 448.4094 1.992660 12.10567 0.1006303 0.2938241 0 153 24  
5 2022-05-31 08:02:03 448.4095 1.992572 12.10567 0.1006321 0.2928492 0 153 24

 

I have added the /functions subfolder here in Oscar's repository: RespChamberProc/main/functions

bgctw commented 3 months ago

"if timesteps are not regular (i.e. if function convertToRegularTimesteps is not applied), the function subsetContiguous gives an error."

Can you, please provide a minimal working example (MWE) of error in subsetContiguous with non-equidistant time steps?

The following MWE works with non-equidistant time steps on current master.

fName <- system.file(
  "genData/SMANIE_Chamber1_26032015.zip", package = "RespChamberProc")
ds0 <- readDat(
  unz(fName, filename = unzip(fName, list = TRUE)[1,"Name"] ),tz = "UTC")
ds <- ds0 %>% 
  filter(UQ(sym("TIMESTAMP")) < as.POSIXct("2015-03-26 06:29:12", tz = "UTC"))
ds$Pa <- ds$AirPres * 100  # convert hPa to Pa
ds$CO2_dry <- corrConcDilution(ds, colConc = "CO2_LI840", colVapour = "H2O_LI840")
ds$H2O_dry <- corrConcDilution(ds, colConc = "H2O_LI840", colVapour = "H2O_LI840")
ds$VPD <- calcVPD( ds$SurTemp, ds$Pa, ds$H2O_LI840)
#
diff(ds$TIMESTAMP)
ds <- mutate(ds, TIMESTAMP = TIMESTAMP + pmin(0.04, pmax(-0.04,rnorm(n(),0.0, 0.2))))
#
dsChunked <- subsetContiguous(ds, colTime = "TIMESTAMP", colIndex = "Collar")
michaelstrupler commented 3 months ago

Hi Thomas,

Please find attached the MWE and the sample data (zipped) we used.

if we don’t interpolate the timesteps and convert them to a regular interval of 1 sec, we get the following error:

Warning message: Unknown or uninitialised column: iChunk.

head(dsChunk)

A tibble: 0 × 1

ℹ 1 variable: iChunk

Thanks and best regards, Michael

 

Michael Strupler, Dr. Sc.

@.*** https://michaelstrupler.quarto.pub/

On 10 Jun 2024, at 16:28, Thomas Wutzler @.***> wrote:

"if timesteps are not regular (i.e. if function convertToRegularTimesteps is not applied), the function subsetContiguous gives an error."

Can you, please provide a minimal working example (MWE) of error in subsetContiguous with non-equidistant time steps?

The following MWE works with non-equidistant time steps on current master.

fName <- system.file( "genData/SMANIE_Chamber1_26032015.zip", package = "RespChamberProc") ds0 <- readDat( unz(fName, filename = unzip(fName, list = TRUE)[1,"Name"] ),tz = "UTC") ds <- ds0 %>% filter(UQ(sym("TIMESTAMP")) < as.POSIXct("2015-03-26 06:29:12", tz = "UTC")) ds$Pa <- ds$AirPres * 100 # convert hPa to Pa ds$CO2_dry <- corrConcDilution(ds, colConc = "CO2_LI840", colVapour = "H2O_LI840") ds$H2O_dry <- corrConcDilution(ds, colConc = "H2O_LI840", colVapour = "H2O_LI840") ds$VPD <- calcVPD( ds$SurTemp, ds$Pa, ds$H2O_LI840) # diff(ds$TIMESTAMP) ds <- mutate(ds, TIMESTAMP = TIMESTAMP + pmin(0.04, pmax(-0.04,rnorm(n(),0.0, 0.2)))) # dsChunked <- subsetContiguous(ds, colTime = "TIMESTAMP", colIndex = "Collar")

— Reply to this email directly, view it on GitHub https://github.com/bgctw/RespChamberProc/issues/7#issuecomment-2158524134, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ3TPVN3WGCNTYG2AJNMFWLZGWZ2TAVCNFSM6AAAAABI23DHXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJYGUZDIMJTGQ. You are receiving this because you authored the thread.

bgctw commented 3 months ago

Dear Michael,

I debugged the code and data that you sent additionally by email. The problem is with argument "gapLength = 1" in 

dsChunk <- subsetContiguous(ds, colTime = "TIMESTAMP", colIndex = "Collar",
                            gapLength = 1, minNRec = 180, minTime = 180, indexNA = 13)

where you tell that differences in timestamps larger than 1 second indicate the start of another contiguous series. The default of 12 seconds is convenient for a chamber that you move around and place it on different collars and initiate another measurement cycle or automatic chambers that only run (an log) each x minutes.

With non-equidistant time steps, there were many changes below 1 second. so that resulting contiguous series were shorter than 180 records or 180s, so that there was no valid contiguous series.

Please, try with an increased value of gapLength, or keep the default of 12 seconds. Then RespChamberProc should work with the original TimeStamps.

Yours Thomas

michaelstrupler commented 3 months ago

Dear Thomas,

Thanks for clarification regarding the increased value of gapLength. Ok, I see, so the function to create continuous timestamps becomes obsolete.

Thanks again, Michael

On 11 Jun 2024, at 15:48, Thomas Wutzler @.***> wrote:

Dear Michael,

I debugged the code and data that you sent additionally by email. The problem is with argument "gapLength = 1" in

dsChunk <- subsetContiguous(ds, colTime = "TIMESTAMP", colIndex = "Collar", gapLength = 1, minNRec = 180, minTime = 180, indexNA = 13) where you tell that differences in timestamps larger than 1 second indicate the start of another contiguous series. The default of 12 seconds is convenient for a chamber that you move around and place it on different collars and initiate another measurement cycle or automatic chambers that only run (an log) each x minutes.

With non-equidistant time steps, there were many changes below 1 second. so that resulting contiguous series were shorter than 180 records or 180s, so that there was no valid contiguous series.

Please, try with an increased value of gapLength, or keep the default of 12 seconds. Then RespChamberProc should work with the original TimeStamps.

Yours Thomas

— Reply to this email directly, view it on GitHub https://github.com/bgctw/RespChamberProc/issues/7#issuecomment-2160815523, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ3TPVITVJLIRTQ73TACR43ZG353LAVCNFSM6AAAAABI23DHXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRQHAYTKNJSGM. You are receiving this because you authored the thread.

michaelstrupler commented 3 months ago

Hi Thomas,

Thought the gapLength should be adapted to the new setup the issue I see is that PICARRO collect data continuously and there is no time off when moving the chamber except the valve (collar) number.

Cheers Oscar

On 11 Jun 2024, at 16:42, Michael Strupler @.***> wrote:

Dear Thomas,

Thanks for clarification regarding the increased value of gapLength. Ok, I see, so the function to create continuous timestamps becomes obsolete.

Thanks again, Michael

On 11 Jun 2024, at 15:48, Thomas Wutzler @.***> wrote:

Dear Michael,

I debugged the code and data that you sent additionally by email. The problem is with argument "gapLength = 1" in

dsChunk <- subsetContiguous(ds, colTime = "TIMESTAMP", colIndex = "Collar", gapLength = 1, minNRec = 180, minTime = 180, indexNA = 13) where you tell that differences in timestamps larger than 1 second indicate the start of another contiguous series. The default of 12 seconds is convenient for a chamber that you move around and place it on different collars and initiate another measurement cycle or automatic chambers that only run (an log) each x minutes.

With non-equidistant time steps, there were many changes below 1 second. so that resulting contiguous series were shorter than 180 records or 180s, so that there was no valid contiguous series.

Please, try with an increased value of gapLength, or keep the default of 12 seconds. Then RespChamberProc should work with the original TimeStamps.

Yours Thomas

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/bgctw/RespChamberProc/issues/7*issuecomment-2160815523__;Iw!!D9dNQwwGXtA!UD1dN_YlfcCJD9Tixbhk9POzeEP8F2A4FPxHxv5nbxx1PZzFijUqhgpy1-mmJurMmSDdZBik2e0M5NU4LAGSa9xKAcDW$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AJ3TPVITVJLIRTQ73TACR43ZG353LAVCNFSM6AAAAABI23DHXWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRQHAYTKNJSGM__;!!D9dNQwwGXtA!UD1dN_YlfcCJD9Tixbhk9POzeEP8F2A4FPxHxv5nbxx1PZzFijUqhgpy1-mmJurMmSDdZBik2e0M5NU4LAGSa7zywy2C$. You are receiving this because you authored the thread.

bgctw commented 3 months ago

Dear Oscar, note the "either" in the function documentation: "to determine these chunks, either a change in an index variable (input by between the measurements) or a gap in time is used."