search

OMP-IRD / hyfaa-scheduler

HYFAA/MGB scheduler designed by Magellium
0 stars 2 forks source link

another error in the fake_previsions step (running 100 ensembles) #3

Open jeanpommier opened 3 years ago

jeanpommier commented 3 years ago

Hi @remijugier , I've run the scheduler on the server, with 100 ensembles as we said. I hoped that on a brand new run, the previ would work. They didn't, but I got a different error: 

Starting fake_previsions_using_previous_years_forcing ...
Database has not been initialized => this must be done in write mode...
Traceback (most recent call last):
  File "/hyfaa/bin/fake_previsions_using_previous_years_forcing.py", line 328, in <module>
    fake_previsions_using_previous_years_forcing(args.input_yaml_file, args.ndays, output_dir=args.output_dir, nyearsmax=args.nyearsmax, nprocs=args.nprocs, verbose=args.verbose)
  File "/hyfaa/bin/fake_previsions_using_previous_years_forcing.py", line 104, in fake_previsions_using_previous_years_forcing
    with HydroStates_DBManager(dico['hydrological_states_database_directory'], mode='r', verbose=verbose) as hydrostates_db_in:
  File "/hyfaa/lib/python3.7/site-packages/hyfaa/database/hydrostates/hydrostates_db.py", line 50, in __init__
    super().__init__(db_sql_path, mode=mode, verbose=verbose)
  File "/hyfaa/lib/python3.7/site-packages/hyfaa/database/basic_filehandler_db.py", line 58, in __init__
    self.__initialization__()
  File "/hyfaa/lib/python3.7/site-packages/hyfaa/database/basic_filehandler_db.py", line 134, in __initialization__
    self._check_writable_()
  File "/hyfaa/lib/python3.7/site-packages/hyfaa/database/basic_filehandler_db.py", line 97, in _check_writable_
    self._check_within_context_()
  File "/hyfaa/lib/python3.7/site-packages/hyfaa/database/basic_filehandler_db.py", line 92, in _check_within_context_
    raise Exception('Object must be within a context')
Exception: Object must be within a context

Any idea how to solve it ?

AdrienHydroMatters commented 3 years ago

same error here with the basic configuration

vpimag commented 3 years ago

Adrien, It looks like you are running the fake_prevision program from an empty database. Could you look at the database_manager.sql file which is stored in the mgbstandard_solution_databases/hydrological_states_db directory and tell me if it is empty?

AdrienHydroMatters commented 3 years ago

Nope, it is 200Ko worth! (-rw-r--r-- 1 root root 200704 Apr 15 09:16 database_manager.sql), and created when I execute Hyfaa (see date and hour)..

vpimag commented 3 years ago

Ok. I reproduced your error by removing the database_manager.sql file so it really looks like it cannot find the database for a reason. Have you checked the consistency of the path written in the yaml file (lines below)?

hydrological state database : used as an input and an output to see which date was treated first

hydrological_states_database_directory: ${hyfaa_workdir}/mgbstandard_solution_databases/hydrological_states_db

Otherwise, you can send me your database file, and the last nc file in your results directory as well as the yaml file (if you modified it).

AdrienHydroMatters commented 3 years ago

out_hyfaa.zip

vpimag commented 3 years ago

I've used your data and it worked for me so I'm afraid the problem might be related with your Windows environment and I'm not the best interlocutor to help you on that. Apparently, Jean faced the same issue at some point. Are you both using a docker?

AdrienHydroMatters commented 3 years ago

Yes, docker image in a Debian sub-system for windows...

jeanpommier commented 3 years ago

AS reported in the issue, I have this problem when running hyfaa from the server (Debian 10), using the docker image. Only change: I've changed the nb of ensembles to 100, as it was decided.

Hyfaa through the docker image is currently working well on my computer (Debian 10 too).

The only differences I see between the server and my computer:

jeanpommier commented 3 years ago

@vpimag have you tried it from scratch ? @AdrienOceanNext was your run started from scratch ?

if @vpimag -> no and @AdrienOceanNext -> yes, we might have an interesting lead

AdrienHydroMatters commented 3 years ago

If I'm right the is a "pre-computed database" in the docker image, non? If yes, mine is not exactly from scratch, as I just installed and ran the solution "as is" (but I already tried to make a all fresh run from scratch, did not worked either; and I did not tried again after the fix from @vpimag #2...)

remijugier commented 3 years ago

AS reported in the issue, I have this problem when running hyfaa from the server (Debian 10), using the docker image. Only change: I've changed the nb of ensembles to 100, as it was decided.

Hyfaa through the docker image is currently working well on my computer (Debian 10 too).

The only differences I see between the server and my computer:

  • 20 ensembles on my computer, 100 on the server
  • I had already run the scheduler before, on my computer (first run dates back to ddfda5f I believe), while on the server, the first run relied on commit 78b44a7 I have not tried running the whole process from scratch on my computer, given the amount of data it downloads.

Hi, some quick info just in case :

remijugier commented 3 years ago

I've just tried running it on my side and it works. Since this issue seems environment related, I suggest that you try running ./run_docker.sh bash and then check that you can access mgbstandard_solution_databases/hydrological_states_db and same for assimilated and ensemblist hydrological state databases

vpimag commented 3 years ago

If by "from scratch" you mean with no previous hydrological state, yes I dit it when I fixed issue #2, and no when I tried to run the code from Adrien's results.
Starting the run from scratch or not should not be an issue as long as there is a previous state to be found when reaching the prevision step. Also, it is not important that you didn't start from scratch after fix #2. I suggest using sqlitebrowser to help you manipulate the sql databases. If you want to start the run from a specific hydrological state, you just have to remove the entries after this date. For example, try to remove the 2 last dates and run it again even if I don't see why it would make a difference. I think there might be some sort of right issue, because your database is there but it is not found.

jeanpommier commented 3 years ago

Thank @remijugier . It would be worth documenting somewhere

Running the process using docker should avoid environment-related issues. And on the server, I'm using the same environment than on my local computer (Debian 10, docker). In both cases, I first downloaded the 2 files as instructed in https://github.com/OMP-IRD/hyfaa-scheduler/blob/master/work_configurations/operational_niger_gsmap/README.md The only differences I see are the nb of ensembles,n and the version of the code when I first ran the processes

jeanpommier commented 3 years ago

@remijugier

Question: if I've already done a run, but I want to change the nb of ensembles, what should I remove ? Only databases/hydrological_states_db and *_solution_databases folders ?

vpimag commented 3 years ago

Change ensemble size :

Start from scratch from a specific date :

Start from specific previous hydrological state :

In all these cases, you can keep assimilation and forcing db (if they do not change of course).

jeanpommier commented 3 years ago

@vpimag thanks

I must say I've never seen a mini_gtp_ensemble file or folder anywhere

vpimag commented 3 years ago

Ok. Then the folder referred as the folder store for the perturbed static data in the yaml file (see lines below): perturb_static_data: activate: true type: normal mode: per_variable folder_store: ${hyfaa_workdir}/mgbstandard_solution_databases/mini_gtp_ensemble

vpimag commented 3 years ago

@jeanpommier you need to activate the perturbation function to get a perturbed ensemble (required for data assimilation)

jeanpommier commented 3 years ago

By default, it is not activated. I'm supposed to activate it ?

remijugier commented 3 years ago

@jeanpommier you need to activate the perturbation function to get a perturbed ensemble (required for data assimilation)

@vpimag I did not activate this option by default so assimilation is only done on the perturbations of hydrological state due to the perturbation of the rain ensemble.

remijugier commented 3 years ago

@jeanpommier you need to activate the perturbation function to get a perturbed ensemble (required for data assimilation)

@vpimag I did not activate this option by default so assimilation is only done on the perturbations of hydrological state due to the perturbation of the rain ensemble.

@vpimag if you want to set different paremeters for @jeanpommier then the best option IMO is that you push new versions of the configuration files on the git repo so @jeanpommier does not have to do it on his end "manually"

remijugier commented 3 years ago

@remijugier

Question: if I've already done a run, but I want to change the nb of ensembles, what should I remove ? Only databases/hydrological_states_db and *_solution_databases folders ?

everything you need to delete to start modelling from scratch without having to download everythin again is in *_solution_databases folders, you should not have a databases/hydrological_states_db unless left over by an old run when it used to be configured this way. For instance if you change ensemble size for the ensemblist case then just delete ensemblist_solution_databases. Same goes for the assimilation case

jeanpommier commented 3 years ago

@remijugier @vpimag I've run the model from scratch on my computer (Debian 10, docker 20.10.5) and ended out with the same error. "From scratch" meaning:

So, given into consideration that:

I'd say it is not environment-related.

Could it be related to the most recent changes, i.e. backporting the changes from Laetitia ?

Comment: I'd rather not have to manipulate the DB's content. On the server, I won't have access to any fancy interface. If there are manipulations to run on the DBs, it would be better to have them documented as commandline commands. And commandline commands sort of document themselves, better that descriptions of what to do in a GUI BTW, .sql is not a classic sqlite extension. Usually, .sql files are for textual SQL representation (DB dumps, queries etc). It is a bit confusing.

vpimag commented 3 years ago

Quickly on the comment, the thing is you are both (Jean and Adrien) running hyfaa but for different purposes (I assumed). So, my remark on how to manipulate DBs was not adressed to Jean (you don't need this kind of manipulation in an operational context) but to Adrien who, I assumed, wants to use it for research purposes (and in that case you might need a way to quickly manipulate your DBs to try out several configurations). Sorry if it was confusing.

AdrienHydroMatters commented 3 years ago

To answer to Jean's question: I'm on Debian 10 (WSL2) And Vanessa: eventually this will be for research, for now I am just trying to get a complete output (assim, standard, previ) :)

Just tried activating the perturbation, will tell you the resultt after it runs.

vpimag commented 3 years ago

Adrien, from what I saw you are running hyfaa with only one member no? If it's the case, no need to activate the perturbation of course.

AdrienHydroMatters commented 3 years ago

Well, yes, for the mgb_standard_solution. There I did not activated perturbation, am I right? For the ensemblist & assimilated solution I did activated it.