Calling add-event for an empty project fails with "Attempting to overwrite existing event. "

[lcocault-cap]

Description

While testing the RAMP framework with the "Titanic" challenge from the "ramp kits", in order to check the proper setup of my environment, my "add-event" command is rejected with a message "Attempting to overwrite existing event. " whereas my database is actually empty. Note that, in my context, the PostgreSQL database is deployed from a Docker image. I do not see any for that deployment detail to cause the issue, but it is probably the only "specific" deployment decision I have made.

Steps/Code to Reproduce

RAMP installation is based on the following steps:

1. python3.7 -m venv ramp-board
2. ramp-board/bin/activate
3. pip install --trusted-host artifactory.cnes.fr -r requirements.txt
4. make install
6. createuser -P ramp
7. createdb -O ramp rampdb
8. pip install --trusted-host artifactory.cnes.fr -r requirements.txt
9. ramp_test_submission --submission starting_kit --save-y-preds
10. ramp-test --submission random_forest_20_5 --save-y-preds
11. ramp_leaderboard --metric=acc
12. mkdir ramp-challenge
13. cd ramp-challenge/
14. ramp-setup init
15. ramp-setup init-event --name=titanic_bateau_coule
16. Edit the events/titanic_bateau_coule/config.yml (to point to the database)
17. cd events/titanic_bateau_coule
18. cp -R /work/RAMP/titanic/* .
19. ramp-database add-event

Expected Results

I expected the add-event to succeed

Actual Results

Instead, I have the following stacktrace: Traceback (most recent call last): File "/work/RAMP/venv/ramp-board/bin/ramp-database", line 11, in load_entry_point('ramp-database==0.9.0.dev0', 'console_scripts', 'ramp-database')() File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_database/cli.py", line 540, in start main() File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1128, in call return self.main(args, kwargs) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, ctx.params) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 754, in invoke return __callback(args, **kwargs) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_database/cli.py", line 261, in add_event force, File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_database/tools/event.py", line 261, in add_event "Attempting to overwrite existing event. " ValueError: Attempting to overwrite existing event. Use force=True to overwrite.

When I check my database, I can confirm there is no existing event in the "events" table.

Anyway, I tried running the command with the force argument as suggested: ramp-database add-event --force=True Traceback (most recent call last): File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2634, in delete state = attributes.instance_state(instance) AttributeError: 'list' object has no attribute '_sa_instance_state'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/work/RAMP/venv/ramp-board/bin/ramp-database", line 11, in load_entry_point('ramp-database==0.9.0.dev0', 'console_scripts', 'ramp-database')() File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_database/cli.py", line 540, in start main() File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1128, in call return self.main(args, kwargs) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, ctx.params) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 754, in invoke return __callback(args, **kwargs) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_database/cli.py", line 261, in add_event force, File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_database/tools/event.py", line 264, in add_event delete_event(session, event_name) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_database/tools/event.py", line 72, in delete_event session.delete(event) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 2638, in delete replacecontext=err, File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise raise exception sqlalchemy.orm.exc.UnmappedInstanceError: Class 'builtins.list' is not mapped

Versions

Python 3.7.12 (default, Nov 17 2021, 09:09:30) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] NumPy 1.21.4 SciPy 1.7.3 Scikit-Learn 1.0.1

[agramfort]

thx @lcocault-cap for the detailed bug report.

@frcaud can you try to replicate?

@lcocault-cap can you share the docker file you used to setup the container?

[lcocault-cap]

@agramfort , since I test within the CNES environment I depend on the images available in the CNES artifactory and I am not sure it is relevant to extract and send the whole image file. However, based on the image metadata, I think the version I use corresponds to this commit: https://github.com/docker-library/postgres/commit/7f5f6da5a1976bfd2c6d989e20cef080d0d9c68f

[frcaud]

@lcocault-cap did you first try to deploy the event with the 'deploy-event' command ?

Deploy event (add problem and event to the database, optionally clone kit and data)

ramp-setup deploy-event [OPTIONS]

It seems to me that add-event should be used to add an event to an existing problem (+event) previously deployed with 'deploy-event'

[lcocault-cap]

I tried the deploy-event and I have another error.

Traceback (most recent call last): File "/work/RAMP/venv/ramp-board/bin/ramp-setup", line 11, in load_entry_point('ramp-utils==0.9.0.dev0', 'console_scripts', 'ramp-setup')() File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_utils/cli.py", line 234, in start main() File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1128, in call return self.main(args, kwargs) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1659, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, ctx.params) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/click/core.py", line 754, in invoke return __callback(args, **kwargs) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_utils/cli.py", line 131, in deploy_event deploy.deploy_ramp_event(config, event_config, cloning, force) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_utils/deploy.py", line 36, in deploy_ramp_event ramp_config = generate_ramp_config(event_config, config) File "/work/RAMP/venv/ramp-board/lib/python3.7/site-packages/ramp_utils/ramp.py", line 62, in generate_ramp_config config = event_config["ramp"] KeyError: 'ramp'

Maybe something is missing in my configuration file?

[rth]

ramp-database add-event

@lcocault-cap Can you try:

ramp setup deploy-event --event-config events/iris_test/config.yml --no-cloning

instead as indicated in https://paris-saclay-cds.github.io/ramp-docs/ramp-board/stable/create_ramp_event.html#deploy-a-specific-ramp-event

I don't know if anyone used ramp-database add-event recently, but personally, I never did to create new events.

[lcocault-cap]

Hey, this is far better. After a few configuration actions, it finally worked. Now, my events table is initialized:

rampdb=# select * from events; 3	titanic_bateau_coule	Human readable event name to display on website	3	1	80	f	f	t	t	f	900	2000-01-01 00:00:00	2100-01-01 00:00:00	2100-01-01 00:00:00	auc
0
starting_kit	/work/RAMP/venv/ramp-board/ramp-challenge/events/titanic_bateau_coule/submissions

As a conclusion, what do you advice for this issue? Should I close it as irrelevant? Or is there a problem with the add-event that I am the first to declare because this feature would never be used? Maybe, the add-event action should be disposed?

[agramfort]

+1 to deprecate "add-event action"

Message ID: @.***>

[frcaud]

I tested 'add-event' further. It creates an event in database when the options are filled and provided that problem (+event) was previously created with deploy-event: Example: ramp setup init-event --name titanic_test (fill config.yml with relevant infos) ramp setup deploy-event --event-config events/titanic_test/config.yml Now titanic_test is deployed and the event appears in the DB as well as on frontend

If we now use add-event for titanic_test2: ramp-database add-event --problem titanic --event titanic_test2 --title <title> --submissions-dir <subdir> this new event will appear in the DB but not on frontend.

Best way to create another event for an already deployed problem is to use deploy-event though: first init-event: ramp setup init-event --name titanic_test3 (fill config.yml with relevant infos) ramp setup deploy-event --event-config events/titanic_test3/config.yml

titanic_test and titanic_test3 are correctly deployed on frontend and appear in DB titanic_test2 is on DB but not on frontend

[agramfort]

thanks for the investigation. Do not hesitate to update the doc to clarify this.

Message ID: @.***>