thouska / spotpy

A Statistical Parameter Optimization Tool
https://spotpy.readthedocs.io/en/latest/
MIT License
248 stars 150 forks source link

ValueError in FAST sampler, reading CSV ("X columns instead of Y") #271

Open arthur-e opened 3 years ago

arthur-e commented 3 years ago

I'm trying to run a small test (i.e., small number of samples) of FAST so as to become more familiar with how it is implemented here.

# cal_process is an instance of the class that has "evaluation" and "simulation" methods, etc.
fast = spotpy.algorithms.fast(cal_process, dbname = 'RECO', dbformat = 'csv')
with warnings.catch_warnings():
    warnings.simplefilter('ignore')
    fast.sample(10)

This same dataset and sampling scheme works perfectly well with ordinary MC (spotpy.algorithms.mc()).

However, FAST fails to produce any meaningful results due to an apparent issue with reading the CSV ("database")...

Initializing the  Fourier Amplitude Sensitivity Test (FAST)  with  10  repetitions
Starting the FAST algotrithm with 10 repetitions...
Creating FAST Matrix
Initialize database...
['csv', 'hdf5', 'ram', 'sql', 'custom', 'noData']
* Database file 'RECO.csv' created.
2 of 10, min objf=1e+308, max objf=-1e+308, time remaining: 00:00:15
3 of 10, min objf=1e+308, max objf=-1e+308, time remaining: 00:00:20
4 of 10, min objf=1e+308, max objf=-1e+308, time remaining: 00:00:19
5 of 10, min objf=1e+308, max objf=-1e+308, time remaining: 00:00:17
6 of 10, min objf=1e+308, max objf=-1e+308, time remaining: 00:00:14
7 of 10, min objf=1e+308, max objf=-1e+308, time remaining: 00:00:10
8 of 10, min objf=1e+308, max objf=-1e+308, time remaining: 00:00:05
9 of 10, min objf=1e+308, max objf=-1e+308, time remaining: 00:00:00
10 of 10, min objf=1e+308, max objf=-1e+308, time remaining: 23:59:55
11 of 10, min objf=1e+308, max objf=-1e+308, time remaining: 23:59:49
12 of 10, min objf=1e+308, max objf=-1e+308, time remaining: 23:59:44

*** Final SPOTPY summary ***
Total Duration: 75.33 seconds
Total Repetitions: 12
Minimal objective value: 1e+308
Corresponding parameter setting:
CUE: nan
tsoil: nan
smsf0: nan
smsf1: nan
Maximal objective value: -1e+308
Corresponding parameter setting:
CUE: nan
tsoil: nan
smsf0: nan
smsf1: nan
******************************

Traceback (most recent call last):
  File "scripts/spot.py", line 259, in <module>
    fire.Fire(CalibrationCLI)
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 681, in _CallAndUpda
teTrace
    component = fn(*varargs, **kwargs)
  File "scripts/spot.py", line 251, in tune_reco_sensitivity
    fast.sample(samples)
  File "/usr/local/pythonenv/smap-l4c/lib/python3.8/site-packages/spotpy/algorithms/fast.py", line 237, in sample
    data = self.datawriter.getdata()
  File "/usr/local/pythonenv/smap-l4c/lib/python3.8/site-packages/spotpy/database/csv.py", line 61, in getdata
    data = np.genfromtxt(self.dbname + '.csv', delimiter=',', names=True)
  File "/usr/lib/python3/dist-packages/numpy/lib/npyio.py", line 2089, in genfromtxt
    raise ValueError(errmsg)
ValueError: Some errors were detected !
    Line #2 (got 361 columns instead of 592111)
    Line #3 (got 1 columns instead of 592111)
    Line #4 (got 1 columns instead of 592111)
    ...
    Line #738601 (got 2 columns instead of 592111)

Where ... indicates I've truncated the error output, above. It's a huge number of output lines (about 738,601).

thouska commented 3 years ago

I cannot reproduce this behavior, could you provide a small test case, where this particular error is happening (preferable by using one of the examples)?