Enable custom pdb upload to RHS

[Waztom]

@kaliif implemented changes to accommodate RHS upload to Fragalysis V2 - see #1327.

@boriskovar-m2ms confirms the uploading of custom pdb files to the RHS will require several days of effort. Summary from Slack chat below.

Review for Yellow release.

unfortunately to make custom pdb work will take several days of work (maybe even up to full week). I thought that I'm one bug (well in this case it's not a bug because it was not implemented and using lhs pdbs was implemented and we were told not to worry about custom pdbs because it's not clear how even backend is going to handle them) away but I now see almost whole picture. Whole picture is displaying data, save/restore of the snapshot, iterating using arrows where dataset has mixed type of compounds (i.e. some of them are using pdbs from lhs and some of them have their own pdbs).Some refactoring is in order to ensure that beyond some point we don't need worry about origin of the pdb to lower the complexity of the code.

[boriskovar-m2ms]

It might take more than one week because I found out that usage of LHS pdb was not implemented for Selected Compounds list. It's different list than list of compounds of one dataset.

[mwinokan]

• F/E still needs lots of work
• B/E is serving custom PDB in the API call
• CSET upload does not yet support the custom PDB uploads

@mwinokan to talk to @kaliif about the work needed

[mwinokan]

I tried uploading a test dataset (to v2 production) today but I get the following error:

MW_2A_P129_custom_pdb.sdf.zip

P129_custom_pdb.zip

Traceback (most recent call last): File "/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 581, in get_or_create return self.get(**kwargs), False ^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 435, in get raise self.model.DoesNotExist( viewer.models.ComputedSetSubmitter.DoesNotExist: ComputedSetSubmitter matching query does not exist. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "viewer_computedsetsubmitter_name_method_09cdfd9c_uniq" DETAIL: Key (name, method)=(Max Winokan, Fragmenstein) already exists. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task R = retval = fun(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/celery.py", line 306, in _inner reraise(*exc_info) File "/.venv/lib/python3.11/site-packages/sentry_sdk/_compat.py", line 115, in reraise raise value File "/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/celery.py", line 301, in _inner return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__ return self.run(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/code/viewer/tasks.py", line 104, in process_compound_set compound_set = save_mols.task() ^^^^^^^^^^^^^^^^ File "/code/viewer/cset_upload.py", line 590, in task mols_to_process = self.set_descriptions( ^^^^^^^^^^^^^^^^^^^^^^ File "/code/viewer/cset_upload.py", line 491, in set_descriptions computed_set.submitter = self.get_submission_info(description_mol) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/code/viewer/cset_upload.py", line 461, in get_submission_info return ComputedSetSubmitter.objects.get_or_create( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.11/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 588, in get_or_create return self.create(**params), True ^^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 453, in create obj.save(force_insert=True, using=self.db) File "/.venv/lib/python3.11/site-packages/django/db/models/base.py", line 739, in save self.save_base(using=using, force_insert=force_insert, File "/.venv/lib/python3.11/site-packages/django/db/models/base.py", line 776, in save_base updated = self._save_table( ^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.11/site-packages/django/db/models/base.py", line 881, in _save_table results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.11/site-packages/django/db/models/base.py", line 919, in _do_insert return manager._insert( ^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.11/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.11/site-packages/django/db/models/query.py", line 1270, in _insert return query.get_compiler(using=using).execute_sql(returning_fields) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql cursor.execute(sql, params) File "/.venv/lib/python3.11/site-packages/sentry_sdk/integrations/django/__init__.py", line 641, in execute result = real_execute(self, sql, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 66, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers return executor(sql, params, many, context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 79, in _execute with self.db.wrap_database_errors: File "/.venv/lib/python3.11/site-packages/django/db/utils.py", line 90, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/.venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ django.db.utils.IntegrityError: duplicate key value violates unique constraint "viewer_computedsetsubmitter_name_method_09cdfd9c_uniq" DETAIL: Key (name, method)=(Max Winokan, Fragmenstein) already exists.

/viewer/upload_task/601a6e03-1b0b-4c89-86cb-ad0d10d37e5d/

[mwinokan]

@boriskovar-m2ms has ensured that the RHS buttons for [L] [P] and [C] buttons work, as well as the navigation with the arrows. Then the changes will need to be ported to the 'Selected Compounds' tab.

Boris estimates that it will take up to another 5 days of work.

[mwinokan]

@boriskovar-m2ms can you try with the following data for CHIKV_Mac please:

CHIKV_JA_2.zip

N.B. I spotted an issue with the previous zip

[mwinokan]

@boriskovar-m2ms please check this CHIKV_JA_3.zip

[mwinokan]

@boriskovar-m2ms is blocked because he does not have test data.

@mwinokan to prepare an A71EV2A dataset

@boriskovar-m2ms is also fixing various RHS issues as he discovers them, but please only fix low-hanging fruit

[mwinokan]

@boriskovar-m2ms please try this RHS dataset for A71EV2A

2a_test_data_20240409.zip

[boriskovar-m2ms]

@mwinokan when I try to use your RHS dataset with target provided by @tdudgeon I get following errors:

So I tried to patch the .sdf file to use correct .pdb files and existing LHS observations and also fixed the <ref_mols> fields to contain also existing LHS observations I get following error from backend:

[mwinokan]

@boriskovar-m2ms I see that Tim's upload_1 dataset has 'a' and not 'A' in the config.yaml for the code_prefix. So I understand why the SDF didn't work at first.

The traceback error could be some of the other SDF fields. Please try SDF with less metadata

2a_test_rhs_20240411.zip

This one should work without having to change the ref_mols

[boriskovar-m2ms]

@mwinokan @kaliif when I upload the files in the previous comment (here is the log) and I request the compounds via api call api/compound-molecules (here is the link) you can see that for both compounds both fields site_observation_code and pdb_info are null which is silently crashing the NGL view causing (sometimes more sometimes less) subtle errors because the exception will silently skip some functionality.

Based on the contents of the .sdf file the pdb_info should be populated.

I will also implemented checks if this happens in the wild the dangerous buttons will be disabled.

[boriskovar-m2ms]

If you change one compound to use LHS pose reference then this compound is uploaded correctly but the one using custom pdb is still broken so it seems that only uploading custom pdb in this case doesn't work.

[mwinokan]

@boriskovar-m2ms has now got working RHS data again, it works in the 'compound sets' tab and changes are being ported to 'selected compounds'. Boris says that these two tabs are separate components, maybe something to fix in the future.

[mwinokan]

@boriskovar-m2ms I prepared a RHS upload with 10 compounds for you. Because I didn't actually have 10 realistic compound designs to hand for 2A (quickly) I just used some of the LHS hits, but the formatting should be exactly as if they are new designs. Let me know if it works for you.

2a_test_rhs_20240418.zip

[boriskovar-m2ms]

Modded and fixed version compatible with target A71EVA2 (file name lb32627-66_09-04-2024.tgz). Modded to be mix of custom pdb and references to LHS observations. Fixed to use a prefix instead of A and replaced missing inspiration with observation that exists in aforementioned target/.tgz file.

2a_test_rhs_20240418-modded.zip

Modded previous RHS so now it is possible to test interactions between multiple computed datasets.

2a_test_rhs_20240411-modded.zip

[mwinokan]

@mwinokan to test with A71EV2A on staging