search

xchem / fragalysis-loader

Django backend loader for fragalysis app
Apache License 2.0
1 stars 4 forks source link

'Target' object has no attribute 'metadata' (latest loader/stack combination) #47

Open alanbchristie opened 4 years ago

alanbchristie commented 4 years ago

Seeing this error again with the latest stack (2.0.4).

  1. I load the stack "cold", i.e. a fresh database and clean media directory.
  2. The stack appears to initialise without error.
  3. The stack is running 2 replicas as a StatefulSet (both stack logs attached).

With the stack initialised I run the latest loader with 2020-09-15T16 data. At the end of the load process the following is seen: -

2020-09-25T08:33:33.294905402Z Analysing target: mArh
2020-09-25T08:33:33.306474363Z Analysing 215 molecules for mArh
2020-09-25T08:33:33.324318370Z Traceback (most recent call last):
2020-09-25T08:33:33.325016196Z   File "loader.py", line 43, in <module>
2020-09-25T08:33:33.325026696Z     process_target(prefix, target_name, app=app)
2020-09-25T08:33:33.325030986Z   File "/code/loader/loaders.py", line 1042, in process_target
2020-09-25T08:33:33.325269299Z     analyse_target(target_name, target_path)
2020-09-25T08:33:33.325275409Z   File "/code/loader/loaders.py", line 748, in analyse_target
2020-09-25T08:33:33.325404590Z     target.metadata.save(
2020-09-25T08:33:33.325411890Z AttributeError: 'Target' object has no attribute 'metadata'
2020-09-25T08:33:33.426257177Z Sentry is attempting to send 1 pending error messages
2020-09-25T08:33:33.426286097Z Waiting up to 2.0 seconds
2020-09-25T08:33:33.426289217Z Press Ctrl-C to quit
2020-09-25T08:33:35.097884550Z Generated private data
2020-09-25T08:33:35.149706236Z +> Done.

stack0.log stack1.log s3-loader.log

alanbchristie commented 4 years ago

When the stack's database is inspected we see the following...

root@database-0:/# psql -U fragalysis frag
psql (12.2 (Debian 12.2-2.pgdg100+1))
Type "help" for help.

frag=> \c frag
You are now connected to database "frag" as user "fragalysis".
frag=> \d viewer_target
                                       Table "public.viewer_target"
   Column    |           Type           | Collation | Nullable |                  Default
-------------+--------------------------+-----------+----------+-------------------------------------------
id          | integer                  |           | not null | nextval('viewer_target_id_seq'::regclass)
title       | character varying(200)   |           | not null |
init_date   | timestamp with time zone |           | not null |
uniprot_id  | character varying(100)   |           |          |
metadata    | character varying(255)   |           |          |
zip_archive | character varying(255)   |           |          |
Indexes:
    "viewer_target_pkey" PRIMARY KEY, btree (id)
    "viewer_target_title_key" UNIQUE CONSTRAINT, btree (title)
    "viewer_target_title_d9d4607a_like" btree (title varchar_pattern_ops)
Referenced by:
    TABLE "hotspots_hotspotmap" CONSTRAINT "hotspots_hotspotmap_target_id_id_9592b642_fk_viewer_target_id" FOREIGN KEY (target_id_id) REFERENCES viewer_target(id) DEFERRABLE INITIALLY DEFERRED
    TABLE "hypothesis_targetresidue" CONSTRAINT "hypothesis_targetres_target_id_id_a749feda_fk_viewer_ta" FOREIGN KEY (target_id_id) REFERENCES viewer_target(id) DEFERRABLE INITIALLY DEFERRED
    TABLE "scoring_molgroup" CONSTRAINT "scoring_molgroup_target_id_id_bfe365d2_fk_viewer_target_id" FOREIGN KEY (target_id_id) REFERENCES viewer_target(id) DEFERRABLE INITIALLY DEFERRED
    TABLE "viewer_activitypoint" CONSTRAINT "viewer_activitypoint_target_id_id_05cb6b4c_fk_viewer_target_id" FOREIGN KEY (target_id_id) REFERENCES viewer_target(id) DEFERRABLE INITIALLY DEFERRED
    TABLE "viewer_computedset" CONSTRAINT "viewer_computedset_target_id_3e4aa356_fk_viewer_target_id" FOREIGN KEY (target_id) REFERENCES viewer_target(id) DEFERRABLE INITIALLY DEFERRED
    TABLE "viewer_protein" CONSTRAINT "viewer_protein_target_id_id_6a7638f2_fk_viewer_target_id" FOREIGN KEY (target_id_id) REFERENCES viewer_target(id) DEFERRABLE INITIALLY DEFERRED
    TABLE "viewer_sessionproject" CONSTRAINT "viewer_sessionproject_target_id_3ebe59c3_fk_viewer_target_id" FOREIGN KEY (target_id) REFERENCES viewer_target(id) DEFERRABLE INITIALLY DEFERRED
    TABLE "viewer_target_project_id" CONSTRAINT "viewer_target_project_id_target_id_99796dcd_fk_viewer_target_id" FOREIGN KEY (target_id) REFERENCES viewer_target(id) DEFERRABLE INITIALLY DEFERRED

frag=>

Thought: _Assuming viewer_target is the table in question could this be because the column's there but there's no data?_

alanbchristie commented 4 years ago

When I reproduce the fault and inspect the target object (printing target.__dict__) I see: -

2020-09-26T11:05:26.779718065Z Skipping confidence - mArh-x1092_0_B
2020-09-26T11:05:26.803313076Z Analysing target: mArh
2020-09-26T11:05:26.805968109Z {'init_date': datetime.datetime(2020, 9, 26, 11, 4, 49, 509857, tzinfo=<UTC>), 'uniprot_id': None, '_state': <django.db.models.base.ModelState object at 0x7f3ce8ec3d10>, 'id': 1, 'title': u'mArh'}

Indeed - there is no metadata.

I suspect this is because the corresponding Target model's FileField is defined with null=True...

class Target(models.Model):
    [...]
    metadata = models.FileField(upload_to="metadata/", null=True, max_length=255)

Observation: In the loader there's a target.root_data_directory = relative_to_media_root(target_path). What's that all about? The current Target object has no root_data_directory property

reskyner commented 4 years ago

The media root is left over from something. I’m on holiday until the 5th October, so won’t look at it until then. Maybe Duncan could have a look.

Sent from my iPhone

On 26 Sep 2020, at 14:00, Alan B. Christie notifications@github.com wrote:



When I reproduce the fault and inspect the target object (printing target.dict) I see: -

2020-09-26T11:05:26.779718065Z Skipping confidence - mArh-x1092_0_B 2020-09-26T11:05:26.803313076Z Analysing target: mArh 2020-09-26T11:05:26.805968109Z {'init_date': datetime.datetime(2020, 9, 26, 11, 4, 49, 509857, tzinfo=), 'uniprot_id': None, '_state': <django.db.models.base.ModelState object at 0x7f3ce8ec3d10>, 'id': 1, 'title': u'mArh'}

Indeed - there is no metadata.

I suspect this is because the corresponding Target model's FileField is defined with null=True...

class Target(models.Model): [...] metadata = models.FileField(upload_to="metadata/", null=True, max_length=255)

Observation: In the loader there's a target.root_data_directory = relative_to_media_root(target_path). What's that all about? The current Target object has no root_data_directory property

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/xchem/fragalysis-loader/issues/47#issuecomment-699485966, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ADBMODLVT3FICCNSXROVHNDSHXJTPANCNFSM4RZKP55Q.

-- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

alanbchristie commented 4 years ago

(thanks)

I've patched the fragalysis-backend and built a private image of the stack with the Target.metadata (and Target.zip_archive) set to blank rather than null: -

class Target(models.Model):
    [...]
    # metadatafile containing sites info for download
    metadata = models.FileField(upload_to="metadata/", blank=True, max_length=255)
    # zip archive to download uploaded data from
    zip_archive = models.FileField(upload_to="archive/", blank=True, max_length=255)

With the corresponding migrations adjusted it seems to resolve the issue as the loader appears to progress to completion without significant error (using a target object that has a metadata property)...

2020-09-26T13:01:30.932675067Z Analysing target: mArh
2020-09-26T13:01:30.935429934Z {'_state': <django.db.models.base.ModelState object at 0x7f496fe036a0>, 'id': 1, 'title': 'mArh', 'init_date': datetime.datetime(2020, 9, 26, 13, 0, 52, 662609, tzinfo=<UTC>), 'uniprot_id': None, 'metadata': <FieldFile: None>, 'zip_archive': <FieldFile: None>, '_django_cleanup_original_cache': {'zip_archive': <FieldFile: None>, 'metadata': <FieldFile: None>}}

The loader concludes with apparent success...

2020-09-26T13:02:55.307781094Z 166
2020-09-26T13:02:55.320974654Z 192
2020-09-26T13:02:57.941728384Z <rdkit.Chem.rdchem.Bond object at 0x7f496fdc92b0>
2020-09-26T13:03:17.690627838Z Generated private data
2020-09-26T13:03:17.825247086Z +> Done.

I'll leave it to others to decide the right course of action here but suspect that an adjusted model and a corresponding migration is probably the best?

alanbchristie commented 4 years ago

I notice that the zip_archive (manipulated in the same code block) also suffers from this issue - see code at the end of analyse_target() method of loaders.py: -

    target.zip_archive.name = relative_to_media_root(zipped)

With regard to a solution ... the general pattern in the community is not to use null=True for properties of this type, i.e. the preference is to use blank=True for non-mandatory properties and that will require a new migration (simple, but just wanted to point this out). Happy to provide a PR on the backend but as this appears to be a change to the model style of the backend repo then someone needs to suggest a solution pattern.