search

specify / specify7

Specify 7
https://www.specifysoftware.org/products/specify-7/
GNU General Public License v2.0
66 stars 36 forks source link

`cannot unpack non-iterable NoneType object` when trying to create a DwC Archive export #2920

Open grantfitzsimmons opened 1 year ago

grantfitzsimmons commented 1 year ago

Update 2024-07-26: This particular issue was due to the export being run manually creating the archive as the export is scoped to the current collection.

It seems if you try to manually run an export using the "Create DwC Archive" function in the User Tools menu for a different collection than the one you are currently in, the above error will be encountered.

​I​'ve found that the "cannot unpack non-iterable NoneType object" error can occur due to two possible reasons:

  1. The ExportFeed is referencing an app resource that does not exist.
  2. Attempting to manually create a DwC archive for a collection that is not the logged-in user's current one.

Specify 7 Crash Report - 2023-01-30T21 56 51.421Z.txt

https://sp7demofish.specifycloud.org/specify/overlay/make-dwca/

Select DwCA_Fish and DwCA_Metadata. See error.

image
grantfitzsimmons commented 4 months ago

Encountered this today on https://kufish51623-issue-545.test.specifysystems.org/specify/overlay/make-dwca/

Specify 7 Crash Report - 2024-07-11T01_35_21.253Z.txt

Generated based on the Tissue DwCA export definition and metadata.

DwCA_metadata_and_def.zip

Logs from the specify7 container:

[11/Jul/2024 01:32:51] [DEBUG] [specifyweb.permissions.permissions:58] permissions check: [(PermRequest(collectionid=4, userid=1, resource='/export/dwca', action='execute'), True, [{'collectionid': None, 'userid': 1, 'resource': '%', 'action': '%'}], [{'roleid': 5, 'rolename': 'Collection Admin', 'resource': '%', 'action': '%'}])]
[11/Jul/2024 01:32:51] [INFO] [specifyweb.context.app_resource:57] looking for app resource 'DwCA_tissue' for user spfishadmin in KU Fish Voucher Collection
[11/Jul/2024 01:32:51] [INFO] [specifyweb.context.app_resource:133] looking in DB at level: Personal
[11/Jul/2024 01:32:51] [INFO] [specifyweb.context.app_resource:81] looking in FS at level: Personal
[11/Jul/2024 01:32:51] [INFO] [specifyweb.context.app_resource:133] looking in DB at level: UserType
[11/Jul/2024 01:32:51] [INFO] [specifyweb.context.app_resource:81] looking in FS at level: UserType
[11/Jul/2024 01:32:51] [INFO] [specifyweb.context.app_resource:133] looking in DB at level: Collection
[11/Jul/2024 01:32:51] [INFO] [specifyweb.context.app_resource:81] looking in FS at level: Collection
[11/Jul/2024 01:32:51] [INFO] [specifyweb.context.app_resource:133] looking in DB at level: Discipline
[11/Jul/2024 01:32:51] [INFO] [specifyweb.context.app_resource:81] looking in FS at level: Discipline
[11/Jul/2024 01:32:51] [INFO] [specifyweb.context.app_resource:133] looking in DB at level: Common
[11/Jul/2024 01:32:51] [INFO] [specifyweb.context.app_resource:81] looking in FS at level: Common
[11/Jul/2024 01:32:51] [INFO] [specifyweb.context.app_resource:133] looking in DB at level: Backstop
[11/Jul/2024 01:32:51] [INFO] [specifyweb.context.app_resource:81] looking in FS at level: Backstop
[11/Jul/2024 01:32:51] [ERROR] [django.request:224] Internal Server Error: /export/make_dwca/
Traceback (most recent call last):
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/specify7/specifyweb/specify/views.py", line 34, in wrapped
    return view(request, *args, **kwargs)
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/views/decorators/http.py", line 40, in inner
    return func(request, *args, **kwargs)
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/opt/specify7/specifyweb/export/views.py", line 121, in export
    definition, _, __ = get_app_resource(collection, user, dwca_resource)
TypeError: cannot unpack non-iterable NoneType object
grantfitzsimmons commented 3 months ago

This seems to be happening independent of the actual report definition. When copying over the metadata and DWC export def, it does not happen on other databases. I am testing this issue as it was reported by NHMD, and the logs look just the same:

https://github.com/NHMDenmark/DanSpecify/blob/978589d73bc5ec960304ff63657a71dc3aa276e6/App%20Resources/NHMD/Mammalogy/NHMD%20Mammalogy/DwCA-MM.xml https://github.com/NHMDenmark/DanSpecify/blob/978589d73bc5ec960304ff63657a71dc3aa276e6/App%20Resources/NHMD/Mammalogy/NHMD%20Mammalogy/DwCA-MM.eml.xml

Specify 7 Crash Report - 2024-07-26T22_07_20.519Z.txt

[26/Jul/2024 22:06:10] [ERROR] [django.request:224] Internal Server Error: /export/make_dwca/
Traceback (most recent call last):
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/specify7/specifyweb/specify/views.py", line 34, in wrapped
    return view(request, *args, **kwargs)
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/views/decorators/http.py", line 40, in inner
    return func(request, *args, **kwargs)
  File "/opt/specify7/ve/lib/python3.8/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/opt/specify7/specifyweb/export/views.py", line 121, in export
    definition, _, __ = get_app_resource(collection, user, dwca_resource)
TypeError: cannot unpack non-iterable NoneType object
[26/Jul/2024 22:06:10] "POST /export/make_dwca/ HTTP/1.0" 500 10574

Since this isn't due to the actual content of the export definition and metadata, it seems there is another variable at play that leads to this issue taking place. If you use those definition files on a database (like KU Fish) it works as expected...

grantfitzsimmons commented 3 months ago

Upon further investigation, the first error appears to be that we are in the wrong collection. It seems you can only manually run a DwC export if you can retrieve the collection information from which you are retrieving data.

Once I switched to the NHMD Mammalogy collection, the export worked as expected.

grantfitzsimmons commented 3 months ago

This is not related to the above issue, feel free to ignore:

https://github.com/NHMDenmark/DanSpecify/blob/978589d73bc5ec960304ff63657a71dc3aa276e6/App%20Resources/NHMD/ExportFeed.xml

When updating the RSS Feed from the NHMD Mammalogy collection, I get the following:

[26/Jul/2024 23:12:11] "POST /export/force_update/ HTTP/1.0" 200 2
[26/Jul/2024 23:12:11] [WARNING] [specifyweb.stored_queries.execution:118] query not filtered by scope
[26/Jul/2024 23:12:11] [WARNING] [specifyweb.stored_queries.execution:118] query not filtered by scope
[26/Jul/2024 23:12:12] [WARNING] [specifyweb.stored_queries.execution:118] query not filtered by scope
[26/Jul/2024 23:12:12] [ERROR] [specifyweb.export.views:172] update_feed failed: Traceback (most recent call last):
  File "/opt/specify7/specifyweb/export/views.py", line 169, in try_update_feed
    update_feed(force=True, notify_user=user)
  File "/opt/specify7/specifyweb/export/feed.py", line 55, in update_feed
    dwca_def, _, __ = get_app_resource(
TypeError: cannot unpack non-iterable NoneType object
image

download (7).txt

Resulting RSS feed (localhost/export/rss/):

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<rss xmlns:ipt="http://ipt.gbif.org/" version="2.0">
<channel>
<link>http://localhost/export/rss/</link>
<title>NHMD Collections DarwinCore RSS Feed</title>
<description>RSS Feed for collection data of the Natural Museum of Denmark, Copenhagen</description>
<item>
<title>BHMD Biocultural Botany Collection</title>
<id>2dd8f3fe-da1e-4a97-9e84-9a652845913d</id>
<link>http://localhost/static/depository/export_feed/DwCA-BC.zip</link>
<ipt:eml>http://localhost/export/extract_eml/DwCA-BC.zip</ipt:eml>
<pubDate>Fri, 26 Jul 2024 23:18:56 -0000</pubDate>
<type>DWCA</type>
</item>
<item>
<title>NHMD Danekræ Fossil Trove Collection</title>
<id>d9c16c4c-7102-4783-9225-4eb0ab079eb2</id>
<link>http://localhost/static/depository/export_feed/DwCA-DK.zip</link>
<ipt:eml>http://localhost/export/extract_eml/DwCA-DK.zip</ipt:eml>
<pubDate>Fri, 26 Jul 2024 23:18:56 -0000</pubDate>
<type>DWCA</type>
</item>
<item>
<title>NHMD Herpetology Collection</title>
<id>46055d79-4602-4bc6-99bc-e7036054a4ec</id>
<link>http://localhost/static/depository/export_feed/DwCA-HP.zip</link>
<ipt:eml>http://localhost/export/extract_eml/DwCA-HP.zip</ipt:eml>
<pubDate>Fri, 26 Jul 2024 23:18:57 -0000</pubDate>
<type>DWCA</type>
</item>
</channel>
</rss>

When running the same from the NHMD Ichthyology collection:

[26/Jul/2024 23:14:05] "POST /export/force_update/ HTTP/1.0" 200 2
[26/Jul/2024 23:14:05] [WARNING] [specifyweb.stored_queries.execution:118] query not filtered by scope
[26/Jul/2024 23:14:06] [WARNING] [specifyweb.stored_queries.execution:118] query not filtered by scope
[26/Jul/2024 23:14:06] [WARNING] [specifyweb.stored_queries.execution:118] query not filtered by scope
[26/Jul/2024 23:14:07] [ERROR] [specifyweb.export.views:172] update_feed failed: Traceback (most recent call last):
  File "/opt/specify7/specifyweb/export/views.py", line 169, in try_update_feed
    update_feed(force=True, notify_user=user)
  File "/opt/specify7/specifyweb/export/feed.py", line 55, in update_feed
    dwca_def, _, __ = get_app_resource(
TypeError: cannot unpack non-iterable NoneType object
image

Resulting RSS feed (localhost/export/rss/):

<rss xmlns:ipt="http://ipt.gbif.org/" version="2.0">
<channel>
<link>http://localhost/export/rss/</link>
<title>NHMD Collections DarwinCore RSS Feed</title>
<description>RSS Feed for collection data of the Natural Museum of Denmark, Copenhagen</description>
<item>
<title>BHMD Biocultural Botany Collection</title>
<id>2dd8f3fe-da1e-4a97-9e84-9a652845913d</id>
<link>http://localhost/static/depository/export_feed/DwCA-BC.zip</link>
<ipt:eml>http://localhost/export/extract_eml/DwCA-BC.zip</ipt:eml>
<pubDate>Fri, 26 Jul 2024 23:14:06 -0000</pubDate>
<type>DWCA</type>
</item>
<item>
<title>NHMD Danekræ Fossil Trove Collection</title>
<id>d9c16c4c-7102-4783-9225-4eb0ab079eb2</id>
<link>http://localhost/static/depository/export_feed/DwCA-DK.zip</link>
<ipt:eml>http://localhost/export/extract_eml/DwCA-DK.zip</ipt:eml>
<pubDate>Fri, 26 Jul 2024 23:14:06 -0000</pubDate>
<type>DWCA</type>
</item>
<item>
<title>NHMD Herpetology Collection</title>
<id>46055d79-4602-4bc6-99bc-e7036054a4ec</id>
<link>http://localhost/static/depository/export_feed/DwCA-HP.zip</link>
<ipt:eml>http://localhost/export/extract_eml/DwCA-HP.zip</ipt:eml>
<pubDate>Fri, 26 Jul 2024 23:14:06 -0000</pubDate>
<type>DWCA</type>
</item>
</channel>
</rss>

Just taking note of this. Seems the successful exports are irrelevant of this failure...

grantfitzsimmons commented 3 months ago

Summary: After some further further investigation, this particular issue was due to the export being run manually creating the archive as the export is scoped to the current collection.

This issue was encountered by NHMD, but it was in pursuit of discovering an unrelated issue due to the DwCA ExportFeed app resource being improperly configured (the feed referenced app resources that did not exist).