search

ibot-geoecology / myClim

R package for processing microclimatic data
GNU General Public License v2.0
6 stars 3 forks source link

Failure in first step #14

Open MichelleDBG opened 2 months ago

MichelleDBG commented 2 months ago

trying to use "mc_read_files" and get the following errors: 

  It isn't possible to read datetimes from ...
  All formats failed to parse. No formats found.   
  File is skipped.

Of the three files pulled by Lolly, I tried to load the "data..." and not the "binary..." or "command_...". I installed 'myClim' from CRAN. I'm running R-4.2.2.

martin-kopecky commented 2 months ago

Dear MichelleDBG,

please, can you share with us one of your files or at least the few first rows of the csv data file? Without seeing the data it is hard to tell where is the problem, but I guess it will be related to the different format of the date-time. Unfortunately, (and for the reasons which are far beyond my imagination), TOMST now allows the user to set in Lolly basically any data-time format :(

MichelleDBG commented 2 months ago

Here is one of my files. I only set the dataformat_name = "TOMST" in trying to run mcread...

Thank you!

Michelle

Michelle DePrenger-Levin, Ph.D. Population Biology Research Associate (she, her, hers) O 720-865-3630 A 909 York Street, Denver, CO 80206 E @.*** W botanicgardens.org VISIT York Street Chatfield Farms Mount Goliath Plains Conservation Center [cid:Artboard1_cfd31e82-9fca-443b-a649-11559710646b.png] [cid:AAM-logo-gray-125x75_196dcba1-ad74-45b4-95bd-6e4610f7477f.png] [cid:APGA-logo-gray-160x75_a67f7c9c-0dba-4d5a-87c4-38a6561a0d7f.png] [cid:SCFD-logo-gray-140x75_c11bad99-6f50-4ce0-878a-dcbb6d4e80da.png]

From: martin-kopecky @.> Sent: Tuesday, July 9, 2024 2:22 PM To: ibot-geoecology/myClim @.> Cc: Michelle DePrenger-Levin @.>; Author @.> Subject: Re: [ibot-geoecology/myClim] Failure in first step (Issue #14)

Dear MichelleDBG,

please, can you share with us one of your files or at least the few first rows of the csv data file? Without seeing the data it is hard to tell where is the problem, but I guess it will be related to the different format of the date-time. Unfortunately, (and for the reasons which are far beyond my imagination), TOMST now allows the user to set in Lolly basically any data-time format :(

- Reply to this email directly, view it on GitHubhttps://url.usb.m.mimecastprotect.com/s/DLibCwn6ZWT7Z7QHVBJXM?domain=github.com, or unsubscribehttps://url.usb.m.mimecastprotect.com/s/NB9MCxoW9XHyqy0fvQgHp?domain=github.com. You are receiving this because you authored the thread.Message ID: @.**@.>>

martin-kopecky commented 2 months ago

Hi Michelle,

you forgot to attach the file :)

martin

MichelleDBG commented 2 months ago

data_95135950_2024_06_24_0.csv data_95135951_2024_06_24_0.csv

MichelleDBG commented 2 months ago

Sorry, I learned that attaching to the email doesn't mean uploading to GitHub.

manmatej commented 2 months ago

Hello Michelle,

In your files, I see three issues.

  1. In your date column, you can see a strange pattern at midnight. image This is one of the known issues of Lolly software. I am not exactly sure which version caused this, but it was something around 1.42. For any further downloads of TOMS TMS, be sure to use the most up-to-date version of Lolly, which is now 1.51.

  2. Your date column is not in the standard output format. Did you change the format yourself in Lolly for some purpose? If you change the date format, you get a warning about the standard format. So it is a good idea to keep the standard. However, it sometimes happens that your date column is broken by Excel. Once you open your TMS data in Excel and then click save (even if you do not change anything), Excel will break your dates and put them exactly in that format you have. Excel is a really bad friend of TMS CSV files. It is always better to load in myClim the data which was never touched by Excel (make a copy for browsing in Excel).

image

  1. The decimal separator in your files is a comma "," which is not the default and is not recommended because it can be easily confused with column separators in CSV. This can again be the work of excel. Or maybe you changed the default in Lolly options?

So, the solution?

The best way would be to report to TOMST and ask them for help at tomst@tomst.com or at least let them know you had the issue with missing hours at midnight. But to be fair, I have a script to solve this because I also suffered from this issue. You have to rebuild your CSV files in R, fixing the date and decimal separator. After this treatment, you can easily read data in myClim.

tms.df <- read.csv("./data1/data_95135950_2024_06_24_0.csv",
header = FALSE, sep = ";", dec = ",")

dt <- do.call("c", lapply(tms.df$V2, as.POSIXct, tryFormats = c("%d.%m.%Y %H:%M:%S","%d.%m.%Y"), tz = "UTC"))
tms.df$V2 <- format.Date(dt, format = "%Y.%m.%d %H:%M")
tms.df <- tms.df[,1:9]
write.table(tms.df, "./data1/data_95135950_2024_06_24_1.csv", sep = ";",
            col.names = FALSE, row.names = FALSE, quote = FALSE, dec = ".")

library(myClim)
tms <- mc_read_files("./data1/data_95135950_2024_06_24_1.csv",
                   dataformat_name = "TOMST")

Regards Matej

MichelleDBG commented 2 months ago

The default for the Time format was yyyy/mm/dd hh:nn. I have changed it to yyyy.mm.dd hh:nn. The default was also "," as the Decimal separator. What should I change it to? Period "." or semicolon ";"?

Is there a way to export new CSVs from lolly with these changes that I won't open in Excel and will only try to upload with myClim?

I reported the errors around midnight to TOMST. I updated the software after I returned from collecting the data from the dataloggers. The code you provided allowed me to read the files. I'll now move on to the rest of your package. Thanks for the help and for providing the myClim package.

Thanks, Michelle

Michelle DePrenger-Levin, Ph.D. Population Biology Research Associate (she, her, hers) O 720-865-3630 A 909 York Street, Denver, CO 80206 E @.*** W botanicgardens.org VISIT York Street Chatfield Farms Mount Goliath Plains Conservation Center [cid:Artboard1_cfd31e82-9fca-443b-a649-11559710646b.png] [cid:AAM-logo-gray-125x75_196dcba1-ad74-45b4-95bd-6e4610f7477f.png] [cid:APGA-logo-gray-160x75_a67f7c9c-0dba-4d5a-87c4-38a6561a0d7f.png] [cid:SCFD-logo-gray-140x75_c11bad99-6f50-4ce0-878a-dcbb6d4e80da.png]

From: Matěj Man @.> Sent: Friday, July 12, 2024 2:53 AM To: ibot-geoecology/myClim @.> Cc: Michelle DePrenger-Levin @.>; Author @.> Subject: Re: [ibot-geoecology/myClim] Failure in first step (Issue #14)

Hello Michelle,

In your files, I see three issues.

  1. In your date column, you can see a strange pattern at midnight. image.png (view on web)https://url.usb.m.mimecastprotect.com/s/4sD8CrgWXOSpOW9U7FBmK?domain=github.com This is one of the known issues of Lolly software. I am not exactly sure which version caused this, but it was something around 1.42. For any further downloads of TOMS TMS, be sure to use the most up-to-date version of Lolly, which is now 1.51.
  2. Your date column is not in the standard output format. Did you change the format yourself in Lolly for some purpose? If you change the date format, you get a warning about the standard format. So it is a good idea to keep the standard. However, it sometimes happens that your date column is broken by Excel. Once you open your TMS data in Excel and then click save (even if you do not change anything), Excel will break your dates and put them exactly in that format you have. Excel is a really bad friend of TMS CSV files. It is always better to load in myClim the data which was never touched by Excel (make a copy for browsing in Excel).

image.png (view on web)https://url.usb.m.mimecastprotect.com/s/28o5Cvm6DVTGZRXtXu2jE?domain=github.com

  1. The decimal separator in your files is a comma "," which is not the default and is not recommended because it can be easily confused with column separators in CSV. This can again be the work of excel. Or maybe you changed the default in Lolly options?

So, the solution?

The best way would be to report to TOMST and ask them for help at @.**@.> or at least let them know you had the issue with missing hours at midnight. But to be fair, I have a script to solve this because I also suffered from this issue. You have to rebuild your CSV files in R, fixing the date and decimal separator. After this treatment, you can easily read data in myClim.

tms.df <- read.csv("./data1/data_95135950_2024_06_24_0.csv",

header = FALSE, sep = ";", dec = ",")

dt <- do.call("c", lapply(tms.df$V2, as.POSIXct, tryFormats = c("%d.%m.%Y %H:%M:%S","%d.%m.%Y"), tz = "UTC"))

tms.df$V2 <- format.Date(dt, format = "%Y.%m.%d %H:%M")

tms.df <- tms.df[,1:9]

write.table(tms.df, "./data1/data_95135950_2024_06_24_1.csv", sep = ";",

        col.names = FALSE, row.names = FALSE, quote = FALSE, dec = ".")

library(myClim)

tms <- mc_read_files("./data1/data_95135950_2024_06_24_1.csv",

               dataformat_name = "TOMST")

Regards Matej

- Reply to this email directly, view it on GitHubhttps://url.usb.m.mimecastprotect.com/s/GMNsCwn6ZWT7n2pS9BLQE?domain=github.com, or unsubscribehttps://url.usb.m.mimecastprotect.com/s/5rH_CxoW9XHyjzOHwQnoo?domain=github.com. You are receiving this because you authored the thread.Message ID: @.**@.>>

martin-kopecky commented 2 months ago

Hi Michelle,

Glad that you are now able to load your data from TMS loggers to myClim. Unfortunately, the new lolly option, which allows the user to select practically any date-time format (and decimal separator as well) is a nightmare for us. We tried to explain what is the problem to people in TOMST, but unsuccessfully (...at least they put the warning in new lolly versions...).

So please, ALWAYS use the original standard format of date-time (yyyy/mm/dd hh:nn) and period "." as a decimal separator.

Regarding the missing time for the midnight measurement - this is a known bug of several lolly version (we reported it immediately to TOMST and also on our Twitter account (see https://x.com/SpaceTimeEco/status/1747567084073005184.). As far as I know, it is not possible to correct these data in lolly (you can either correct the data with the script suggested by Matej, or download the data from the TMS logger again with the new (hopefully bug free) lolly version.

Best regards,

martin

manmatej commented 2 months ago

Hello Michelle,

There is a way to export new CSVs from Lolly. However, the only adjustment that can be made is the conversion/fix of the datetime format. You cannot change the decimal separator ex post (but this is not a problem for myClim, as you can keep the decimal comma ","). myClim can read this; it's just not very safe if you share your CSV with someone whose computer has a different default column separator. The safest approach is to use a semicolon as the column separator (which is the default in Lolly and cannot be changed) and a dot "." as the decimal separator (this can be changed and defaults to your system setting, usually a dot in Europe and a comma in America).

So, how do you fix dates with Lolly?

  1. Ensure your CSV files are in a separate folder containing only data from loggers. Make sure this folder is on the same hard drive where Lolly is installed (if you have multiple hard drives in your computer).
  2. Open Lolly and, in the Options tab, select the data format you wish to use (yyyy/mm/dd hh:nn). Keep this as the default. Note that slashes will be translated to dots (don't know why, but it works).
  3. Return to Lolly and press the "Show Data" button.
  4. In the left panel (browser), navigate to your data folder.
  5. At the top in Lolly, check the "Convert Folder" option to fix all CSVs in the folder.
  6. Press the "Fix Time" button.
  7. This will create new CSV data files in your folder with the prefix "conv_".

image