Can't run code due to date parsing issue

[danielbwright]

Ashar, I tried to run RainyDay_Py3.py on the CONUS404 data and got SystemExit: We could not parse a file date in YYYYMMDD format from the filenames. I thought you said that you'd modified the code to handle this, but that doesn't seem to be true. Can you tell me what's going on here?

[Ashar25]

I didn't fix this issue in the code, instead, I changed the format of the file names in the RAINPATH folder. These are some minor issues that popped up when I tried to run the CONUS 404. So, I just dropped the idea of using it in my project. We actually never talked about this issue because it didn't cross my mind when we met. I can fix that issue if you want me to do that.

[danielbwright]

You told me today in our meeting that you had a regex fix to this problem to read from filenames that had dates in YYYY-MM-DD format, as well as using YYYY/MM/DD. But I don't see this.

[Ashar25]

Oh! No I told this for my current objective i.e., reading files for creating one storm per file. I said we could do the same for fixing the "createfilelist" function.

[danielbwright]

Do you have code for this? If so, your or I should integrate that into createfilelist().

[Ashar25]

Not right now. Give me today and I will push it.

[Ashar25]

The thing is we are expecting the user to provide a YYYYMMDD format. Do you want me to remove that expectation and make it to read all the files including YYYY-MM-DD format too?

[danielbwright]

Yes. Because we need to be able to read the CONUS404 dataset.

[Ashar25]

Ok, let me see what we can do about it.

[danielbwright]

Ashar, it looks like you can use xarray.open_mfdataset to read in only the desired variable (e.g. 'time') using the data_vars argument: https://docs.xarray.dev/en/stable/generated/xarray.open_mfdataset.html. So it should be possible to use your plan of rewriting createfilelist() using that functionality.

[Ashar25]

This argument works only for variables and not coordinates (i.e, time here). The thing is I want to decide on what is expected from this function. Currently, this function is 1) checking if the file format is correct, 2) returns the number of years included, and 3) the list which contains the required files after excluding the months and including the years. As far as I have understood this doesn't require dealing with any dataset rather, we are focusing on which dataset to open. So, this function may not require xarray. Let us be clear on what to do with this function so that the output of this function is compatible with the other functions. First and foremost problem is to make this function read all kinds of files including YYYY-MM-DD and YYYY/MM/DD. Let me know if we require any other formats to be compatible here.

[danielbwright]

Please provide a reproducible example showing that the storm identification is somehow "overselecting" earlier years.

[danielbwright]

Ashar, I agree, that makes sense. Let’s make sure it can read those formats you listed. In the future we will also check YYYYMM, but let us wait to do that till later. Please change to be able to read the other formats you listed. Dan

-- Daniel B. Wright (he/him/his) Associate Professor, Civil and Environmental Engineering University of Wisconsin-Madison Hydroclimate Extremes Research Group (https://her.cee.wisc.edu/)

Visiting Scientist, Office of Water Prediction National Oceanic and Atmospheric Administration

From: Ashar25 @.> Date: Tuesday, June 6, 2023 at 9:47 PM To: Ashar25/RainyDay @.> Cc: Daniel Wright @.>, Author @.> Subject: Re: [Ashar25/RainyDay] Can't run code due to date parsing issue (Issue #2)

This argument works only for variables and not coordinates (i.e, time here). The thing is I want to decide on what is expected from this function. Currently, this function is 1) checking if the file format is correct, 2) returns the number of years included, and 3) the list which contains the required files after excluding the months and including the years. As far as I have understood this doesn't require dealing with any dataset rather, we are focusing on which dataset to open. So, this function may not require xarray. Let us be clear on what to do with this function so that the output of this function is compatible with the other functions. First and foremost problem is to make this function read all kinds of files including YYYY-MM-DD and YYYY/MM/DD. Let me know if we require any other formats to be compatible here.

— Reply to this email directly, view it on GitHubhttps://github.com/Ashar25/RainyDay/issues/2#issuecomment-1579785157, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AECZV4KDBZQ4A63QROO65NTXJ7TUFANCNFSM6AAAAAAY4YO32Y. You are receiving this because you authored the thread.Message ID: @.***>