Error reading csv's: "There aren't any valid localities"

[DonScott1]

I am new to Myclim. I have downloaded data from several TMS-4 sensors and am trying to read them with Myclim.

When I enter:

tms.f <- mc_read_files("data_94247100_2024_01_10_0.csv", dataformat_name = "TOMST", silent = T)

I get:

Error in .read_get_output_data(files_table, localities, data_formats) :
There aren't any valid localities. In addition: Warning message: In .f(filename = .l[[1L]][[i]], data_format = .l[[2L]][[i]], serial_number = .l[[3L]][[i]], : It isn't possible to read datetimes from data_94247100_2024_01_10_0.csv. File is skipped.

I have made a files table and localities table as per the instructions and run:

tms.m <- mc_read_data(files_table = "files_table.csv",

• localities_table =lt,
• silent = T,clean = F)

But get: Error in .read_get_output_data(files_table, localities, data_formats) :
There aren't any valid localities. In addition: There were 50 or more warnings (use warnings() to see the first 50)

I have attached an example data csv and the tables. data_94247031_2024_01_10_0.csv files_table.csv localities_table.csv

Full code I am using below (only pulled from the Myclim documentation)

install.packages("rlang")

requiered_packages <- c("stringr", "lubridate", "tibble", "dplyr", "purrr", "ggplot2", "ggforce", "viridis", "data.table", "rmarkdown", "knitr", "kableExtra", "tidyr", "plotly", "zoo", "vroom") missing_packages <- requiered_packages[!(requiered_packages %in% installed.packages()[,"Package"])] if(length(missing_packages)) install.packages(missing_packages)

optional package rTubeDB

remotes::install_github('environmentalinformatics-marburg/tubedb/rTubeDB')

installation of myClim package

install.packages("http://labgis.ibot.cas.cz/myclim/myClim_latest.tar.gz", repos=NULL, build_vignettes=TRUE)

library(myClim)

setwd("F:/Lincoln/Miscanthus/Miscanthus Field Surveys 2023/TOMST_RD_23")

read from csv files

tms.f <- mc_read_files("data_94247076_2024_01_02_0.csv", dataformat_name = "TOMST", silent = T)

ft<-read.table("files_table.csv",sep=",",header = T) lt<-read.table("localities_table.csv",sep=",",header = T)

tms.m <- mc_read_data(files_table = "files_table.csv", localities_table =lt, silent = T,clean = F)

Please help as getting the required measures without Myclim will be painful!

Thanks

Donald Scott

[Jules-]

According to your file, it appears that TOMST has changed the output format. Our last data are in the same format, so we will need to update the data reading process in myClim to adapt this change. Thank you for bringing this to our attention.

previous format is

55;2022.11.01 00:00;4;7.625;6.0625;4.875;2537;202;0;
56;2022.11.01 00:15;4;7.625;6.0625;5;2537;202;0;
57;2022.11.01 00:30;4;7.5625;6.125;5.25;2537;202;0;
58;2022.11.01 00:45;4;7.5625;6.125;5.25;2537;202;0;
59;2022.11.01 01:00;4;7.5625;6.1875;5.375;2537;202;0;
60;2022.11.01 01:15;4;7.5;6.125;5.375;2537;202;0;
61;2022.11.01 01:30;4;7.5;6.25;5.625;2537;202;0;
62;2022.11.01 01:45;4;7.5;6.125;5.125;2537;202;0;
63;2022.11.01 02:00;4;7.5;6;4.875;2537;202;0;
64;2022.11.01 02:15;4;7.5;6;5;2537;202;0;
65;2022.11.01 02:30;4;7.5;5.875;4.75;2537;202;0;

new format is

0;08.06.2023;4;16.9375;17.375;17;383;202;0;
1;08.06.2023 00:15:00;4;16.9375;17.375;17;383;202;0;
2;08.06.2023 00:30:00;4;16.9375;17.3125;17;383;202;0;
3;08.06.2023 00:45:00;4;16.9375;17.3125;17;383;202;0;
4;08.06.2023 01:00:00;4;16.875;17.3125;17;382;202;0;
5;08.06.2023 01:15:00;4;16.875;17.3125;17;382;202;0;
6;08.06.2023 01:30:00;4;16.875;17.3125;17;382;202;0;
7;08.06.2023 01:45:00;4;16.875;17.25;16.9375;382;202;0;
8;08.06.2023 02:00:00;4;16.875;17.25;16.9375;382;202;0;
9;08.06.2023 02:15:00;4;16.875;17.25;16.875;382;202;0;

[DonScott1]

Hi Jules. Thanks for letting me know. Do you have any idea of the timeframe for amending the package?

Thanks

Donald

[martin-kopecky]

Dear DonScott1,

This was likely a bug of some recent TOMST lolly software versions (probably starting with version 1.42). New versions of lolly (at least from 1.45) does not produce this format. Since it was a bug and not a permanent feature, we will not implement a new functionality to correct this directly into myClim, but we will upload the separate R code which will correct these problematic data (hopefully within a week or so). We will let you know.

[Akronix]

Hello.

I'm facing the same problem here, I was wondering if myClim could add the date_format argument for TOMSTDataFormat as well?, so this could be easily fixed just by adding date_format = "%d.%m.%y %H:%M:%S" to the call to mc_read_files() function; plus it will be also more resilient for new changes in the future of date format from TOMST or more adaptable to different locales.

There would be still a small issue there, which is that midnight datetime removes the 00:00:00 part in the bugged output. I wrote a very simple python script to fix this: https://gist.github.com/Akronix/28c5df0f8db7d6c3cebbbfc5475cf9c9

[martin-kopecky]

Hi Akronix,

We are indeed considering to add parameter date_format, but the issue is more complex, since TOMST now (in lolly 1.45) allows the user to specify anything in date_format (you can even put there only year, and lolly will give you csv file with only years :) Beside this being nonsense, the user-specified data_format is directly against our philosophy during the development of TMS loggers, and we will try to convince the TOMST to remove this option. The user-specified date/time format is hampering any future data reuse and large-scale microclimatic syntheses (given the missing header in TOMST data files), which were one of our main goal during TMS development....

[Akronix]

Hi @martin-kopecky . Thank you for your explanation. I understand that having inconsistent data files, as I do now, it's annoying and troublesome. Plus particularly, in my case, I was not aware of the change and I don't know how to set-up lolly to back up to the old date format. Recently, I was considering writing another script to convert from new to old datetime format (or viceversa), but it's a very time consuming task and I'm trying to avoid it and stick to trait both format.

Anyhow, from our part, I think that at least having the option to provide strptime datetime format for the so-called mc_read_files() would cover most of the cases, doesn't it?

[manmatej]

Hi, to add my two cents... There is also a very TOMST way (undocumented, undescribed, unintuitive) how to solve the issue with nonstandard date formatting ex-post (convert existing csvs) using TOMST Lolly.

1. it is first necessary to specify the date format you wish in Lolly Options panel. The default, myClim friendly date format is yyyy/mm/dd hh:nn

3. then go back to Info and click Show data - this opens TOMST Log view

4. then activate file viewer and navigate to the folder with the csvs you want to convert and click Fix/convert time

5. sometimes it is necessary to click some files in file viewer or to click "show" multiple times and it will convert all data csvs in current folder into the date format you set in Lolly options. It will produce files with "conv" prefix.

[martin-kopecky]

Matej, you forgot to mention, that you need to close lolly after changing the options. Otherwise, the change will have no effect. Indeed, very TOMST way :))

[Jules-]

With version 1.0.17 of myClim, it's now possible to read the TOMST data format using a custom date format. This can even be used to read corrupted format from Lolly 1.42. You can use the following R code for this:

tms.f <- mc_read_files("data_94247031_2024_01_10_0.csv", dataformat_name = "TOMST", date_format = c("%d.%m.%Y %H:%M:%S", "%d.%m.%Y"), silent = T)