retro nwm 3.0 url bug

[JordanLaserGit]

This conf_nwmurl.json

{
  "forcing_type": "retrospective",
  "start_date": "202001200100",
  "end_date": "202001200100",
  "urlbaseinput": 4,
  "selected_object_type": [
    1
  ],
  "selected_var_types": [
    6
  ],
  "write_to_file": true
}

yields this retro_filenamelist.txt

https://noaa-nwm-retrospective-3-0-pds.s3.amazonaws.com/CONUS/netcdf/forcing/2020/2020012001.LDASIN_DOMAIN1

but I think it should be

https://noaa-nwm-retrospective-3-0-pds.s3.amazonaws.com/CONUS/netcdf/FORCING/2020/202001200100.LDASIN_DOMAIN1

so there's just two missing digits for the minutes, not sure why they are required.

[jameshalgren]

Good catch. @RohanSunkarapalli Can you address?

[RohanSunkarapalli]

Yes @jameshalgren , that's right. I gave that condition because when I observed the bucket we saw that the files from year 2007 doesn't have "00" appended to the date format whereas the files from 90's to 2006 has "00" at the tail of the date format.

I gave this condition -

https://github.com/CIROH-UA/nwmurl/blob/a3846bab6322a93ff5aa174ec93c258024b09f05/nwmurl/urlgennwm.py#L488

But if you think it's not needed anymore it could be corrected by modifying the code to this way -

def generate_url_retro(date, file_type, urlbase_prefix, retrospective_var_types=None): year_txt = date.strftime("%Y") date_txt = date.strftime("%Y%m%d%H%M") # Updated to include minutes

if "forcing" in file_type and date.year < 2007:
    url = f"{urlbase_prefix}{file_type}{year_txt}/{date_txt}.LDASIN_DOMAIN1"
elif "forcing" in file_type and date.year >= 2007:
    url = f"{urlbase_prefix}{file_type}{year_txt}/{date_txt}.LDASIN_DOMAIN1"
elif "model_output" in file_type:
    url = [
        f"{urlbase_prefix}{file_type}{year_txt}/{date_txt}{type}"
        for type in retrospective_var_types
    ]

if urlbase_prefix == "https://ciroh-nwm-zarr-retrospective-data-copy.s3.amazonaws.com/noaa-nwm-retrospective-2-1-zarr-pds/":
    for url in url: 
        url = url + ".json"
        url = url.replace('.comp', '')
return url