Issues after commit https://github.com/ArcticSnow/TopoPyScale/commit/5dc0f5772e47cc1b0b710d715658251190389620

[joelfiddes]

some issues after this commit @paswyss can you check my prposed changes below and open Q - thanks:

https://github.com/ArcticSnow/TopoPyScale/commit/5dc0f5772e47cc1b0b710d715658251190389620

Downscaling LW, SW for point: 9.0
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/joel/anaconda3/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/joel/anaconda3/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/home/joel/src/TopoPyScale/TopoPyScale/topo_scale.py", line 501, in pt_downscale_radiations
    down_pt.to_netcdf(f'{downscaled_directory}/{file_pattern.replace("*", pt_id)}',
TypeError: replace() argument 2 must be str, not numpy.float64
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/joel/sim/tscale_projects/fc_basins/b1/run_worker.py", line 61, in <module>
    mp.downscale_climate()
  File "/home/joel/src/TopoPyScale/TopoPyScale/topoclass.py", line 517, in downscale_climate
    ta.downscale_climate(self.config.project.directory,
  File "/home/joel/src/TopoPyScale/TopoPyScale/topo_scale.py", line 513, in downscale_climate
    multicore_pooling(pt_downscale_radiations, fun_param, n_core)
  File "/home/joel/src/TopoPyScale/TopoPyScale/topo_scale.py", line 78, in multicore_pooling
    pool.starmap(fun, fun_param)
  File "/home/joel/anaconda3/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/home/joel/anaconda3/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/home/joel/anaconda3/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/joel/anaconda3/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/home/joel/src/TopoPyScale/TopoPyScale/topo_scale.py", line 501, in pt_downscale_radiations
    down_pt.to_netcdf(f'{downscaled_directory}/{file_pattern.replace("*", pt_id)}',
TypeError: replace() argument 2 must be str, not numpy.float64
(downscaling) joel@mountainsense:~/sim/tscale_projects/fc_basins/b1$ 

I have added str(pt_id) to this linedown_pt.to_netcdf(f'{downscaled_directory}/{file_pattern.replace("*", pt_id)}',

Also I get this as output - why the additional zero? Is it desired behaviour? I have to make some commits to deal with this if its a new convention:

/home/joel/sim/tscale_projects/fc_basins/b1/sim_2000/outputs/downscaled/down_pt_0.0.nc

[joelfiddes]

I also get this:

 File "/home/caduff/sim/tscale_projects/fc_basins/b1/run_worker.py", line 61, in <module>
    mp.downscale_climate()
  File "/home/caduff/src/TopoPyScale/TopoPyScale/topoclass.py", line 513, in downscale_climate
    ta.downscale_climate(self.config.project.directory,
  File "/home/caduff/src/TopoPyScale/TopoPyScale/topo_scale.py", line 518, in downscale_climate
    multicore_pooling(pt_downscale_radiations, fun_param, n_core)
  File "/home/caduff/src/TopoPyScale/TopoPyScale/topo_scale.py", line 79, in multicore_pooling
    pool.starmap(fun, fun_param)
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/home/caduff/src/TopoPyScale/TopoPyScale/topo_scale.py", line 504, in pt_downscale_radiations
    down_pt.to_netcdf(f'{downscaled_directory}/{file_pattern.split("*")[0]}{num}{file_pattern.split("*")[1]}'$
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/site-packages/xarray/core/dataset.py", line 1946, $
    return to_netcdf(  # type: ignore  # mypy cannot resolve the overloads:(
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/site-packages/xarray/backends/api.py", line 1272, $
    dump_to_store(
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/site-packages/xarray/backends/api.py", line 1319, $
    store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/site-packages/xarray/backends/common.py", line 277$
    self.set_dimensions(variables, unlimited_dims=unlimited_dims)
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/site-packages/xarray/backends/common.py", line 348$
    raise ValueError(
ValueError: Unable to update size for existing dimension'time' (8760 != 8784)

[joelfiddes]

i dont get that when I check out the previous commit - something to do with the file naming i think.....

[paswyss]

Hi @joelfiddes That never occured to me. Can you add additional Info of your config file? Then I will check what I find out with the same configuration.

Did you use an ID_Col for the point IDs? If yes and this is not a string it could be leading to errors -> should be fixed with a new commit I just made. If not -> I need more information about the config to reproduce the error :)

[joelfiddes]

my config:

project:
    name: b1
    description: Downscaling for Sry Darya B1
    authors:
    - Fiddes J.
        - Filhol S.
        - Aalstad K.
    date: Feb 2023
    directory: /home/caduff/sim/tscale_projects/fc_basins/b1/
    start: 2000-01-01
    end: 2022-12-31
    extent:  [42.3058436852906965, 41.2235769463910984,69.9501329079825069, 71.9811673400160998]

    CPU_cores: 4
    climate: era5

    split:
    IO: False   # Flag to split downscaling in time or not
        time: 5        # number of years to split timeline in
        space: None     # NOT IMPLEMENTED

climate:
    era5:
    path: /home/caduff/sim/tscale_projects/naryn_climate/
        product: reanalysis
        timestep: 1H
        plevels: [300, 500,600, 700,800, 850, 900, 1000]
        download_threads: 12

dem:
    file: dem.tif
    epsg: 32642
    horizon_increments: 10
    dem_resol: 500

sampling:
    method: toposub
    points:
    csv_file:

    toposub:
    clustering_method: minibatchkmean
        n_clusters: 10
        random_seed: 2
        clustering_features: {'x':1, 'y':1, 'elevation':1, 'slope':1, 'aspect_cos':1, 'aspect_sin':1, 'svf':1}

toposcale:
    interpolation_method: idw
    pt_sampling_method: nearest
    LW_terrain_contribution: True

outputs:
    variables: all  # list or combination name
    file:
    clean_outputs: False
        clean_FSM: True
        df_centroids: df_centroids.pck
        ds_param: ds_param.nc
        ds_solar: ds_solar.nc
        da_horizon: da_horizon.nc
        landform: landform.tif
        downscaled_pt: down_pt_*.nc

clean_up:
    delete_tmp_dirs: False

[paswyss]

Thanks. Did you tried to re create the df_centroids file? (If it was pre saved this could be the error)

[joelfiddes]

thanks for looking at this @paswyss what if ID_Col not declared at all?

[joelfiddes]

you need my dem and climate to make it reproducible - but hopefully its something simple im doing wrong with new settings....

[paswyss]

thanks for looking at this @paswyss what if ID_Col not declared at all?

Then it should be the sama behaviour as before

[joelfiddes]

actually - i guess you could use any climate and dem to test the config

[joelfiddes]

did you commit this:

I have added str(pt_id) to this line down_pt.to_netcdf(f'{downscaled_directory}/{file_pattern.replace("*", pt_id)}',

[paswyss]

did you commit this:

I have added str(pt_id) to this line down_pt.to_netcdf(f'{downscaled_directory}/{file_pattern.replace("*", pt_id)}',

No.

Because at this point the pt_id should already be a string. But maybe I forgot to add the corresponding part if you using toposub sampling. I'll check.

[paswyss]

@joelfiddes Can you try again? (you need to clean the output dir to test) it works for me now wit toposub

[joelfiddes]

Hey @paswyss still seems to be a float:

`ln: cannot remove ‘/home/caduff/.conda/envs/downscaling/share/gdb/auto-load//home/caduff/.conda/envs/downscaling/lib/libarrow.so.1000.$
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/home/caduff/src/TopoPyScale/TopoPyScale/topo_scale.py", line 501, in pt_downscale_radiations
    down_pt.to_netcdf(f'{downscaled_directory}/{file_pattern.replace("*", pt_id)}',
TypeError: replace() argument 2 must be str, not numpy.float64
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/caduff/sim/tscale_projects/fc_basins/b1/run_worker.py", line 61, in <module>
    mp.downscale_climate()
  File "/home/caduff/src/TopoPyScale/TopoPyScale/topoclass.py", line 518, in downscale_climate
    ta.downscale_climate(self.config.project.directory,
  File "/home/caduff/src/TopoPyScale/TopoPyScale/topo_scale.py", line 513, in downscale_climate
    multicore_pooling(pt_downscale_radiations, fun_param, n_core)
  File "/home/caduff/src/TopoPyScale/TopoPyScale/topo_scale.py", line 78, in multicore_pooling
    pool.starmap(fun, fun_param)
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/caduff/.conda/envs/downscaling/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/home/caduff/src/TopoPyScale/TopoPyScale/topo_scale.py", line 501, in pt_downscale_radiations
    down_pt.to_netcdf(f'{downscaled_directory}/{file_pattern.replace("*", pt_id)}',
TypeError: replace() argument 2 must be str, not numpy.float64
`

[joelfiddes]

needs to be a str here: https://github.com/ArcticSnow/TopoPyScale/blob/d8587020eb981ea65cad778073e553f0346b5170/TopoPyScale/topo_scale.py#L501

[paswyss]

needs to be a str here: https://github.com/ArcticSnow/TopoPyScale/blob/d8587020eb981ea65cad778073e553f0346b5170/TopoPyScale/topo_scale.py#L501

Just to be sure: you newly generated the 'df_centroids' (so not loaded the presaved) and you really run on the last commit? Becausi with the config you have, I am not able to reproduce the error on my computer. (hmmm, strange)

Can you upload a print screen of your df_centroids. The point_id there should be a string already. and pt_id will get loaded from there.

[joelfiddes]

aha, let me try again - i have a kind of strange set up on the cluster here so i may have copied an old centroids file....will check

[joelfiddes]

good now :) thanks for dealing with so quickly @paswyss! I think can close now

[joelfiddes]

good now :) thanks for dealing with so quickly @paswyss! I think can close now

[joelfiddes]

good now :) thanks for dealing with so quickly @paswyss! I think can close now

[paswyss]

hurray :) You're welcome!