search

microsoft / aerial_wildlife_detection

Tools for detecting wildlife in aerial images using active learning
MIT License
226 stars 58 forks source link

upgrade_v1_project script #22

Closed ctorney closed 4 years ago

ctorney commented 4 years ago

Hi Beni I'm having some issues running the upgrade script.

Firstly, the config call here: https://github.com/microsoft/aerial_wildlife_detection/blob/a61bc0780ee6485d8db97452a2a2cc2ab3655ca1/projectCreation/upgrade_v1_project.py#L72 picks up the schema from the v2 settings file, shouldn't this be coming from the v1 file? I had to edit the schema in config/settings.ini to match the v1 schema to get it to run which may not be the right thing to do.

It does run but gives errors in the database updates:

insert or update on table "image_user" violates foreign key constraint "image_user_image_fkey"
DETAIL:  Key (username)=(colin) is not present in table "user".

insert or update on table "annotation" violates foreign key constraint "annotation_username_fkey"
DETAIL:  Key (username)=(colin) is not present in table "user".

null value in column "name" violates not-null constraint
DETAIL:  Failing row contains (cameratraps, null, null, f, E2VweuhpF-RR_paP8M_1TOyc8TP335xRvqZYIFSU91c, t, null, null, null, {"enableEmptyClass": "no", "showPredictions": "yes", "showPredic..., null, null, null, null, f, null, null, null, null, null, t, admin, f, f).

insert or update on table "authentication" violates foreign key constraint "authentication_username_fkey"
DETAIL:  Key (username)=(hugheyl) is not present in table "user".
CONTEXT:  SQL statement "INSERT INTO aide_admin.authentication (username, project, isAdmin)
                    SELECT name, 'cameratraps' AS project, isAdmin FROM cameratraps.user"
PL/pgSQL function inline_code_block line 9 at SQL statement

then finally

insert or update on table "authentication" violates foreign key constraint "authentication_project_fkey"
DETAIL:  Key (project)=(cameratraps) is not present in table "project".
CONTEXT:  SQL statement "INSERT INTO aide_admin.authentication (username, project, isAdmin)
                    SELECT name, 'cameratraps', isAdmin FROM cameratraps.user
                    WHERE name IN (SELECT name FROM aide_admin.user)
                    ON CONFLICT (username, project) DO NOTHING"
PL/pgSQL function inline_code_block line 9 at SQL statement

Project "None" has been converted to AIDE v2 standards. Please do not use a v1 installation on this project anymore.

Thanks

bkellenb commented 4 years ago

Hi Colin,

Oh dear, you are right! The migration script indeed picked up the values from the new config .ini file. Thank you for reporting. I just reworked the script; it should now handle parameter origins truthfully and also does a better job at migrating and registering user accounts with the new v2 system.

Please let me know if any errors persist and I will fix it further. I am thus leaving this issue open for now.

ctorney commented 4 years ago

Thanks Beni - just tried again and I'm still seeing a few errors:

insert or update on table "image_user" violates foreign key constraint "image_user_image_fkey"
DETAIL:  Key (username)=(admin) is not present in table "user".

insert or update on table "annotation" violates foreign key constraint "annotation_username_fkey"
DETAIL:  Key (username)=(Mugisha patrick) is not present in table "user".

WARNING: could not parse settings defined in model settings path ("serengeti/yolo.json")
insert or update on table "project" violates foreign key constraint "project_user_fkey"
DETAIL:  Key (owner)=(admin) is not present in table "user".

insert or update on table "authentication" violates foreign key constraint "authentication_project_fkey"
DETAIL:  Key (project)=(cameratraps) is not present in table "project".
CONTEXT:  SQL statement "INSERT INTO aide_admin.authentication (username, project, isAdmin)
                    SELECT name, 'cameratraps', isAdmin FROM cameratraps.user
                    WHERE name IN (SELECT name FROM aide_admin.user)
                    ON CONFLICT (username, project) DO NOTHING"
PL/pgSQL function inline_code_block line 9 at SQL statement

The script does end with:

Project "cameratraps" has been converted to AIDE v2 standards. Please do not use a v1 installation on this project anymore.

but AIDE doesn't launch. What should be the schema in the new config file - aide_admin or should it be a project schema?

bkellenb commented 4 years ago

Hi Colin,

insert or update on table "image_user" violates foreign key constraint "image_user_image_fkey"
DETAIL:  Key (username)=(admin) is not present in table "user".

insert or update on table "annotation" violates foreign key constraint "annotation_username_fkey"
DETAIL:  Key (username)=(Mugisha patrick) is not present in table "user".

insert or update on table "project" violates foreign key constraint "project_user_fkey"
DETAIL:  Key (owner)=(admin) is not present in table "user".

These messages occurred because the upgrade script tried to apply all changes to the database before inserting the user names of the original project. This resulted in foreign key violations (the database tried to link the user names to the new "aide_admin.user" relation, even though the user names were not yet copied).

insert or update on table "authentication" violates foreign key constraint "authentication_project_fkey"`
Likewise, this occurred because the script tried to add project-user authentication entries before the project was registered in the database.

All of these errors should have been fixed in the latest script.

WARNING: could not parse settings defined in model settings path ("serengeti/yolo.json") This message means that the provided AI model config file in brackets either does not exist or does not adhere to JSON encoding standards.

I now fixed the execution order of all the steps; you may want to try running the script again.

ctorney commented 4 years ago

Hi Beni Seem to be getting closer. I can launch AIDE now but have the following warnings on start-up:

constraint "image_user_image_fkey" for relation "image_user" already exists

constraint "image_user_image_fkey" for relation "image_user" already exists

constraint "image_user_image_fkey" for relation "image_user" already exists

constraint "image_user_image_fkey" for relation "image_user" already exists

constraint "annotation_username_fkey" for relation "annotation" already exists

constraint "annotation_username_fkey" for relation "annotation" already exists

constraint "annotation_username_fkey" for relation "annotation" already exists

constraint "image_user_image_fkey" for relation "image_user" already exists

Then when I launch a the project in AIDE, I'm unable to see images in the interface. One issue I noticed is here https://github.com/microsoft/aerial_wildlife_detection/blob/4718bfb519c53f88eacfd4c31e0969999997a2ff/modules/LabelUI/static/js/dataEntry.js#L319 I don't have the 'files' directory in my path to the images. If I take this out in the code or add in a files directory to the image server then I can get images to load in a new project but still not the upgraded project.

I looked at the difference between a new project and the imported project and the import appears to be having a problem here: https://github.com/microsoft/aerial_wildlife_detection/blob/4718bfb519c53f88eacfd4c31e0969999997a2ff/modules/LabelUI/static/js/dataEntry.js#L1013 with the error: TypeError: window.styles.hoverText is undefined

bkellenb commented 4 years ago

Hi Colin,

It's interesting that those warnings occur. I will update the script for later usages to resolve them. They are nothing to be worried about, though.

I don't have the 'files' directory in my path to the images.

This should be correct; the /files/ fragment only applies to the URL under which the images can be found, and not the actual directory on the file system. It should suffice if you move the images to a sub-folder with the same shortname as the project (i.e., the DB schema name). So if you have the following lines in your new .ini file…

[FileServer]
staticfiles_dir = /local/directory
staticfiles_uri = /

…and your migrated project had the shortname cameratraps, then you can move the images to this folder:

/local/directory/cameratraps/<files go here> Perhaps your browser needs to have the cache emptied to update the URI? This should actually not be necessary.

TypeError: window.styles.hoverText is undefined

Alright, it seems like some of the required styles were not transferred. In the latest version (1.8.200830) this should now be resolved—each time the styles are affected (during loading, saving, and in the migration from v1), they are now auto-completed with defaults wherever needed.

ctorney commented 4 years ago

Hi Beni Thanks - that's all working now. I had to remove the dataServer_uri from the config file to get it to load from the local file system - I guess this should be empty if the fileserver is on same vm? Will this cause problems with the AIController? (everything will be running on the same machine). Before I had

dataServer_uri = http://vmname.uksouth.cloudapp.azure.com:8181/data/

which worked if I took out the /files/ fragment but now changed to 

dataServer_uri =

and it loads the images from /local/directory/data/cameratraps

ctorney commented 4 years ago

Hi @bkellenb One final thing with this script, the line: https://github.com/microsoft/aerial_wildlife_detection/blob/a61bc0780ee6485d8db97452a2a2cc2ab3655ca1/projectCreation/upgrade_v1_project.py#L133 gave an error for me and I had to change to:

modelSettings = json.dumps(json.load(f))

Not sure if this is a bug or a version issue with json

bkellenb commented 4 years ago

Thank you @ctorney,

I had to remove the dataServer_uri from the config file to get it to load from the local file system

For clarification:

Not sure if this is a bug or a version issue with json

this was indeed a bug and has been fixed now.