Unable to parse app metadata

[rileyhales]

All installations of the app store currently do not work. It fails to parse some metadata about the apps. Specifically it can't find some urls. if i'm understanding the code right, its expecting to be found in the app metadata that it gets from conda.

Here's the error it spits out in the console

ERROR:django.request:Internal Server Error: /apps/app-store/get_resources/
Traceback (most recent call last):
  File "/Users/rchales/miniconda3/envs/tethys/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/Users/rchales/miniconda3/envs/tethys/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users/rchales/miniconda3/envs/tethys/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/rchales/miniconda3/envs/tethys/lib/python3.7/site-packages/tethys_apps/decorators.py", line 34, in wrapper
    return controller(request, *args, **kwargs)
  File "/Users/rchales/miniconda3/envs/tethys/lib/python3.7/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/Users/rchales/miniconda3/envs/tethys/lib/python3.7/site-packages/tethys_apps/decorators.py", line 201, in _wrapped_controller
    response = controller_func(request, *args, **kwargs)
  File "/Users/rchales/miniconda3/envs/tethys/lib/python3.7/site-packages/tethys_apps/base/workspace.py", line 424, in wrapper
    return controller(*args, the_workspace, **kwargs)
  File "/Users/rchales/code/tethysapps/tethysapp-tethys_app_store/tethysapp/app_store/controllers.py", line 31, in get_resources
    all_resources = fetch_resources(app_workspace, require_refresh)
  File "/Users/rchales/code/tethysapps/tethysapp-tethys_app_store/tethysapp/app_store/resource_helpers.py", line 70, in fetch_resources
    resource_metadata = process_resources(resource_metadata, app_workspace)
  File "/Users/rchales/code/tethysapps/tethysapp-tethys_app_store/tethysapp/app_store/resource_helpers.py", line 111, in process_resources
    app['metadata']['dev_url'] = license_metadata["url"]
KeyError: 'url'
ERROR:django.channels.server:HTTP GET /apps/app-store/get_resources/ 500 [1.34, 127.0.0.1:53048]

Here's the block of code where its failing.

        try:
            if "license" not in app.get("metadata"):
                raise ValueError

            license_metadata = json.loads(app.get("metadata").get(
                "license").replace("', '", '", "').replace("': '", '": "').replace("'}", '"}').replace("{'", '{"'))
            app['metadata'] = add_if_exists(license_metadata, app['metadata'], [
                'author', 'description', 'license', 'author_email', 'keywords'])
            app['metadata']['dev_url'] = license_metadata["url"]

[rileyhales]

@rfun @danames

[rileyhales]

Current working theory, when the app store needs to refresh its list of available apps its getting metadata from conda in a format it doesn't recognize. The app's code isn't prepared to fail when parsing out metadata that doesn't exist.

Perhaps conda had a change in its api? i think the data the app store code is iterating through comes from the results of

        conda_search_result = subprocess.run(['conda', 'search', "-c", CHANNEL_NAME, "--override-channels",
                                              "-i", "--json"], stdout=subprocess.PIPE)

[rileyhales]

Here's an example of what you get when you run this command right now.

conda search -c tethysapp --override-channels -i --json >> conda_metadata_example_2022-03-07.json

conda_metadata_example_2022-03-07.txt

[rileyhales]

Here's a link to the file where the exception happens https://github.com/BYU-Hydroinformatics/tethysapp-tethys_app_store/blob/master/tethysapp/app_store/resource_helpers.py#L78

And to clarify: by the app store doesn't work right now, i mean that the app opens but stays perpetually with the loading gif that shows at the start and in the console it endlessly prints the 500 error i shared in the first message.

[rfun]

Thanks for reporting this @rileyhales . I will take a look at it today and hopefully have a fix in by tomorrow. Sorry for the trouble

[rileyhales]

No worries. If I've understood/diagnosed this properly, it should be a matter of finding a small change to the command to query the conda channel or an exception handler or something. Sounds like a small patch.

Thanks Rohit.

[danames]

I think the bigger question here is what changed that broke this, and how can we avoid breaking changes in the future? If this was a change on conda then is there anything we can do to track/handle those better?

[rileyhales]

@danames its probably because of something that changed in the conda search command. Thats the only explanation I can think of that would break every installation at the same time and give the same error. I didnt find a change log or release notes or anything about it though.

[msouff]

That's one of the running jokes in the data science community. If it looks like it's conda's fault, it probably is.

On Tue, Mar 8, 2022, 7:31 AM Riley Hales @.***> wrote:

@danames https://github.com/danames its probably because of something that changed in the conda search command. Thats the only explanation I can think of that would break every installation at the same time and give the same error. I didnt find a change log or release notes or anything about it though.

— Reply to this email directly, view it on GitHub https://github.com/BYU-Hydroinformatics/tethysapp-tethys_app_store/issues/23#issuecomment-1061839556, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEEGQB7GVE7BYQRET5RV5XDU65QFVANCNFSM5QEVRZ4A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: <BYU-Hydroinformatics/tethysapp-tethys_app_store/issues/23/1061839556@ github.com>

[rfun]

So the issue is not with Conda. Nothing has changed there. A new app "Flood impact viewer" was added yesterday to the app store. The app store builds the metadata as a JSON object from the setup.py file. All setup.py files are expected to have the URL field since it comes standard with scaffolded apps. However in this case the object looks like this: "{'name': 'release_package', 'version': '1.0.0', 'description': 'This app is a viewer app. It displays flood maps and their corresponding imapct statistics tables.', 'long_description': ''}"

There are two steps that have been taken to resolve this issue:

1. Removed Flood impact viewer from the app store temporarily. I have reached out to Travis Tyler to find the source code to his app so that I can inspect the setup.py file. I verified that Tethys itself hasn't updated its scaffold template for apps so he must have removed it manually.

2.) However, having said that, the app should still be able to handle a missing parameter. In the code I was handling all others except the URL field. A new version of the app store (1.0.9) has been pushed out that handles that error.

It is not critical to update to the new version, but we can if we wish to. The App stores on tethys.byu.edu and tethys-staging seem to work just fine, now that I have removed the problematic app from the app store. I will work with Travis in getting that fixed.

@rileyhales : Thanks so much for the time spent in logging this issue and working through it.

[danames]

You guys are all amazing. Riley thanks for figuring it out, and Rohit thanks for digging deep and coming up with some solutions. I wonder how many installations of the App Store we need to track down and upgrade now…?

On Tue, Mar 8, 2022 at 8:18 PM Rohit Khattar @.***> wrote:

Closed #23 https://github.com/BYU-Hydroinformatics/tethysapp-tethys_app_store/issues/23 .

— Reply to this email directly, view it on GitHub https://github.com/BYU-Hydroinformatics/tethysapp-tethys_app_store/issues/23#event-6206984840, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2PZ5D4S5EL4ACRF6UTIJLU7AJ6ZANCNFSM5QEVRZ4A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: <BYU-Hydroinformatics/tethysapp-tethys_app_store/issue/23/issue_event/6206984840 @github.com>

--

---

Daniel P. Ames, Ph.D., P.E. Professor, Civil & Environmental Engineering,Brigham Young University, Provo, Utah, USA Past-President, International Environmental Modelling & Software Society Editor-in-Chief, Environmental Modelling & Software http://ceen.et.byu.edu/content/dan-ames https://ceen.byu.edu/directory/dan-p-ames http://www.researchgate.net/profile/Daniel_Ames http://scholar.google.com/citations?user=S0GUCeUAAAAJ

---