search

navis-org / navis

Python library for analysis of neuroanatomical data.
https://navis-org.github.io/navis/
GNU General Public License v3.0
83 stars 33 forks source link

caveclient > 5.15.2 crashes microns tutorial #146

Open jakobtroidl opened 8 months ago

jakobtroidl commented 8 months ago

Description I think caveclient version > 5.15.2 introduces a bug in the microns interface tutorial.

To Reproduce

pip install caveclient==5.15.2 # works
pip install caveclient==5.17.2 # throw's an error (see attached)
import navis.interfaces.microns as mi
client = mi.get_cave_client(datastack='cortex65')
client.materialize.get_tables()

Traceback for cavclient==5.17.2 is 

{
    "name": "HTTPError",
    "message": "500 Server Error: 'NoneType' object has no attribute 'pop' for url: https://minnie.microns-daf.com/materialize/api/v3/datastack/minnie65_public_v117/version/117/tables/metadata content:b'{\"code\": 500, \"message\": \"\\'NoneType\\' object has no attribute \\'pop\\'\", \"traceback\": [\"Traceback (most recent call last):\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/flask/app.py\\\\\", line 1516, in full_dispatch_request\\\\n    rv = self.dispatch_request()\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/flask/app.py\\\\\", line 1502, in dispatch_request\\\\n    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/flask_restx/api.py\\\\\", line 403, in wrapper\\\\n    resp = resource(*args, **kwargs)\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/flask/views.py\\\\\", line 84, in view\\\\n    return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs)\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/flask_restx/resource.py\\\\\", line 49, in dispatch_request\\\\n    resp = meth(*args, **kwargs)\\\\n\", \"  File \\\\\"/app/./materializationengine/blueprints/reset_auth.py\\\\\", line 12, in decorated_function\\\\n    return f(*args, **kwargs)\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/middle_auth_client/decorators.py\\\\\", line 265, in decorated_function\\\\n    return f(*args, **kwargs)\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/middle_auth_client/decorators.py\\\\\", line 396, in decorated_function\\\\n    return f(*args, **kwargs)\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/flask_limiter/extension.py\\\\\", line 1186, in __inner\\\\n    R, flask.current_app.ensure_sync(cast(Callable[P, R], obj))(*a, **k)\\\\n\", \"  File \\\\\"/app/./materializationengine/blueprints/client/datastack.py\\\\\", line 80, in wrapper\\\\n    return f(*args, **kwargs)\\\\n\", \"  File \\\\\"/app/./materializationengine/blueprints/client/api2.py\\\\\", line 758, in get\\\\n    ann_table = ann_md.pop(\\\\\"annotation_table\\\\\", None)\\\\n\", \"AttributeError: \\'NoneType\\' object has no attribute \\'pop\\'\\\\n\"]}\\n'",
    "stack": "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n\u001b[0;31mHTTPError\u001b[0m                                 Traceback (most recent call last)\nCell \u001b[0;32mIn[2], line 6\u001b[0m\n\u001b[1;32m      3\u001b[0m client\u001b[38;5;241m.\u001b[39mauth\u001b[38;5;241m.\u001b[39mtoken\n\u001b[1;32m      5\u001b[0m \u001b[38;5;66;03m# Fetch available annotation tables\u001b[39;00m\n\u001b[0;32m----> 6\u001b[0m \u001b[43mclient\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmaterialize\u001b[49m\u001b[38;5;241m.\u001b[39mget_tables()\n\nFile \u001b[0;32m~/Desktop/implicit-neurons/venv/lib/python3.9/site-packages/caveclient/frameworkclient.py:449\u001b[0m, in \u001b[0;36mCAVEclientFull.materialize\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    444\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    445\u001b[0m \u001b[38;5;124;03mA client for the materialization service. See [client.materialize](../client_api/materialize.md)\u001b[39;00m\n\u001b[1;32m    446\u001b[0m \u001b[38;5;124;03mfor more information.\u001b[39;00m\n\u001b[1;32m    447\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    448\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_materialize \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 449\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_materialize \u001b[38;5;241m=\u001b[39m \u001b[43mMaterializationClient\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    450\u001b[0m \u001b[43m        \u001b[49m\u001b[43mserver_address\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlocal_server\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    451\u001b[0m \u001b[43m        \u001b[49m\u001b[43mauth_client\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mauth\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    452\u001b[0m \u001b[43m        \u001b[49m\u001b[43mdatastack_name\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_datastack_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    453\u001b[0m \u001b[43m        \u001b[49m\u001b[43msynapse_table\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43minfo\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_datastack_info\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43msynapse_table\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mNone\u001b[39;49;00m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    454\u001b[0m \u001b[43m        \u001b[49m\u001b[43mmax_retries\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_max_retries\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    455\u001b[0m \u001b[43m        \u001b[49m\u001b[43mpool_maxsize\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_pool_maxsize\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    456\u001b[0m \u001b[43m        \u001b[49m\u001b[43mpool_block\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_pool_block\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    457\u001b[0m \u001b[43m        \u001b[49m\u001b[43mover_client\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m    458\u001b[0m \u001b[43m        \u001b[49m\u001b[43mdesired_resolution\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdesired_resolution\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    459\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    460\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_materialize\n\nFile \u001b[0;32m~/Desktop/implicit-neurons/venv/lib/python3.9/site-packages/caveclient/materializationengine.py:200\u001b[0m, in \u001b[0;36mMaterializationClient\u001b[0;34m(server_address, datastack_name, auth_client, cg_client, synapse_table, api_version, version, verify, max_retries, pool_maxsize, pool_block, desired_resolution, over_client)\u001b[0m\n\u001b[1;32m    188\u001b[0m endpoints, api_version \u001b[38;5;241m=\u001b[39m _api_endpoints(\n\u001b[1;32m    189\u001b[0m     api_version,\n\u001b[1;32m    190\u001b[0m     SERVER_KEY,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    196\u001b[0m     verify\u001b[38;5;241m=\u001b[39mverify,\n\u001b[1;32m    197\u001b[0m )\n\u001b[1;32m    199\u001b[0m MatClient \u001b[38;5;241m=\u001b[39m client_mapping[api_version]\n\u001b[0;32m--> 200\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mMatClient\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    201\u001b[0m \u001b[43m    \u001b[49m\u001b[43mserver_address\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    202\u001b[0m \u001b[43m    \u001b[49m\u001b[43mauth_header\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    203\u001b[0m \u001b[43m    \u001b[49m\u001b[43mapi_version\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    204\u001b[0m \u001b[43m    \u001b[49m\u001b[43mendpoints\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    205\u001b[0m \u001b[43m    \u001b[49m\u001b[43mSERVER_KEY\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    206\u001b[0m \u001b[43m    \u001b[49m\u001b[43mdatastack_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    207\u001b[0m \u001b[43m    \u001b[49m\u001b[43mcg_client\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcg_client\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    208\u001b[0m \u001b[43m    \u001b[49m\u001b[43msynapse_table\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msynapse_table\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    209\u001b[0m \u001b[43m    \u001b[49m\u001b[43mversion\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mversion\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    210\u001b[0m \u001b[43m    \u001b[49m\u001b[43mverify\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mverify\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    211\u001b[0m \u001b[43m    \u001b[49m\u001b[43mmax_retries\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmax_retries\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    212\u001b[0m \u001b[43m    \u001b[49m\u001b[43mpool_maxsize\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mpool_maxsize\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    213\u001b[0m \u001b[43m    \u001b[49m\u001b[43mpool_block\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mpool_block\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    214\u001b[0m \u001b[43m    \u001b[49m\u001b[43mover_client\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mover_client\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    215\u001b[0m \u001b[43m    \u001b[49m\u001b[43mdesired_resolution\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdesired_resolution\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    216\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\nFile \u001b[0;32m~/Desktop/implicit-neurons/venv/lib/python3.9/site-packages/caveclient/materializationengine.py:1894\u001b[0m, in \u001b[0;36mMaterializationClientV3.__init__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1888\u001b[0m     metadata\u001b[38;5;241m.\u001b[39mappend(\n\u001b[1;32m   1889\u001b[0m         executor\u001b[38;5;241m.\u001b[39msubmit(\n\u001b[1;32m   1890\u001b[0m             \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mget_view_schemas\n\u001b[1;32m   1891\u001b[0m         )\n\u001b[1;32m   1892\u001b[0m     )\n\u001b[1;32m   1893\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mfc \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m-> 1894\u001b[0m     tables \u001b[38;5;241m=\u001b[39m TableManager(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mfc, \u001b[43mmetadata\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m]\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mresult\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m, metadata[\u001b[38;5;241m1\u001b[39m]\u001b[38;5;241m.\u001b[39mresult())\n\u001b[1;32m   1895\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m   1896\u001b[0m     tables \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\nFile \u001b[0;32m/usr/local/Cellar/python@3.9/3.9.18/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py:439\u001b[0m, in \u001b[0;36mFuture.result\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m    437\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m CancelledError()\n\u001b[1;32m    438\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_state \u001b[38;5;241m==\u001b[39m FINISHED:\n\u001b[0;32m--> 439\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m__get_result\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    441\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_condition\u001b[38;5;241m.\u001b[39mwait(timeout)\n\u001b[1;32m    443\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_state \u001b[38;5;129;01min\u001b[39;00m [CANCELLED, CANCELLED_AND_NOTIFIED]:\n\nFile \u001b[0;32m/usr/local/Cellar/python@3.9/3.9.18/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/_base.py:391\u001b[0m, in \u001b[0;36mFuture.__get_result\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    389\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception:\n\u001b[1;32m    390\u001b[0m     \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 391\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_exception\n\u001b[1;32m    392\u001b[0m     \u001b[38;5;28;01mfinally\u001b[39;00m:\n\u001b[1;32m    393\u001b[0m         \u001b[38;5;66;03m# Break a reference cycle with the exception in self._exception\u001b[39;00m\n\u001b[1;32m    394\u001b[0m         \u001b[38;5;28mself\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\nFile \u001b[0;32m/usr/local/Cellar/python@3.9/3.9.18/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py:58\u001b[0m, in \u001b[0;36m_WorkItem.run\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     55\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m\n\u001b[1;32m     57\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 58\u001b[0m     result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     59\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mBaseException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m exc:\n\u001b[1;32m     60\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mfuture\u001b[38;5;241m.\u001b[39mset_exception(exc)\n\nFile \u001b[0;32m~/Desktop/implicit-neurons/venv/lib/python3.9/site-packages/cachetools/__init__.py:737\u001b[0m, in \u001b[0;36mcached.<locals>.decorator.<locals>.wrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    735\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m:\n\u001b[1;32m    736\u001b[0m     \u001b[38;5;28;01mpass\u001b[39;00m  \u001b[38;5;66;03m# key not found\u001b[39;00m\n\u001b[0;32m--> 737\u001b[0m v \u001b[38;5;241m=\u001b[39m \u001b[43mfunc\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    738\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m    739\u001b[0m     cache[k] \u001b[38;5;241m=\u001b[39m v\n\nFile \u001b[0;32m~/Desktop/implicit-neurons/venv/lib/python3.9/site-packages/caveclient/materializationengine.py:1939\u001b[0m, in \u001b[0;36mMaterializationClientV3.get_tables_metadata\u001b[0;34m(self, datastack_name, version, log_warning)\u001b[0m\n\u001b[1;32m   1936\u001b[0m url \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_endpoints[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mall_tables_metadata\u001b[39m\u001b[38;5;124m\"\u001b[39m]\u001b[38;5;241m.\u001b[39mformat_map(endpoint_mapping)\n\u001b[1;32m   1938\u001b[0m response \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msession\u001b[38;5;241m.\u001b[39mget(url)\n\u001b[0;32m-> 1939\u001b[0m all_metadata \u001b[38;5;241m=\u001b[39m \u001b[43mhandle_response\u001b[49m\u001b[43m(\u001b[49m\u001b[43mresponse\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mlog_warning\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlog_warning\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1940\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m metadata_d \u001b[38;5;129;01min\u001b[39;00m all_metadata:\n\u001b[1;32m   1941\u001b[0m     vx \u001b[38;5;241m=\u001b[39m metadata_d\u001b[38;5;241m.\u001b[39mpop(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mvoxel_resolution_x\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m)\n\nFile \u001b[0;32m~/Desktop/implicit-neurons/venv/lib/python3.9/site-packages/caveclient/base.py:88\u001b[0m, in \u001b[0;36mhandle_response\u001b[0;34m(response, as_json, log_warning)\u001b[0m\n\u001b[1;32m     86\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mhandle_response\u001b[39m(response, as_json\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m, log_warning\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m):\n\u001b[1;32m     87\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"Deal with potential errors in endpoint response and return json for default case\"\"\"\u001b[39;00m\n\u001b[0;32m---> 88\u001b[0m     \u001b[43m_raise_for_status\u001b[49m\u001b[43m(\u001b[49m\u001b[43mresponse\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mlog_warning\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlog_warning\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     89\u001b[0m     _check_authorization_redirect(response)\n\u001b[1;32m     90\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m as_json:\n\nFile \u001b[0;32m~/Desktop/implicit-neurons/venv/lib/python3.9/site-packages/caveclient/base.py:79\u001b[0m, in \u001b[0;36m_raise_for_status\u001b[0;34m(r, log_warning)\u001b[0m\n\u001b[1;32m     71\u001b[0m     http_error_msg \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m Server Error: \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m for url: \u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m content:\u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m%\u001b[39m (\n\u001b[1;32m     72\u001b[0m         r\u001b[38;5;241m.\u001b[39mstatus_code,\n\u001b[1;32m     73\u001b[0m         reason,\n\u001b[1;32m     74\u001b[0m         r\u001b[38;5;241m.\u001b[39murl,\n\u001b[1;32m     75\u001b[0m         r\u001b[38;5;241m.\u001b[39mcontent,\n\u001b[1;32m     76\u001b[0m     )\n\u001b[1;32m     78\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m http_error_msg:\n\u001b[0;32m---> 79\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m requests\u001b[38;5;241m.\u001b[39mHTTPError(http_error_msg, response\u001b[38;5;241m=\u001b[39mr)\n\u001b[1;32m     80\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m log_warning:\n\u001b[1;32m     81\u001b[0m     warning \u001b[38;5;241m=\u001b[39m r\u001b[38;5;241m.\u001b[39mheaders\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mWarning\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\n\u001b[0;31mHTTPError\u001b[0m: 500 Server Error: 'NoneType' object has no attribute 'pop' for url: https://minnie.microns-daf.com/materialize/api/v3/datastack/minnie65_public_v117/version/117/tables/metadata content:b'{\"code\": 500, \"message\": \"\\'NoneType\\' object has no attribute \\'pop\\'\", \"traceback\": [\"Traceback (most recent call last):\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/flask/app.py\\\\\", line 1516, in full_dispatch_request\\\\n    rv = self.dispatch_request()\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/flask/app.py\\\\\", line 1502, in dispatch_request\\\\n    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/flask_restx/api.py\\\\\", line 403, in wrapper\\\\n    resp = resource(*args, **kwargs)\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/flask/views.py\\\\\", line 84, in view\\\\n    return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs)\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/flask_restx/resource.py\\\\\", line 49, in dispatch_request\\\\n    resp = meth(*args, **kwargs)\\\\n\", \"  File \\\\\"/app/./materializationengine/blueprints/reset_auth.py\\\\\", line 12, in decorated_function\\\\n    return f(*args, **kwargs)\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/middle_auth_client/decorators.py\\\\\", line 265, in decorated_function\\\\n    return f(*args, **kwargs)\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/middle_auth_client/decorators.py\\\\\", line 396, in decorated_function\\\\n    return f(*args, **kwargs)\\\\n\", \"  File \\\\\"/usr/local/lib/python3.9/site-packages/flask_limiter/extension.py\\\\\", line 1186, in __inner\\\\n    R, flask.current_app.ensure_sync(cast(Callable[P, R], obj))(*a, **k)\\\\n\", \"  File \\\\\"/app/./materializationengine/blueprints/client/datastack.py\\\\\", line 80, in wrapper\\\\n    return f(*args, **kwargs)\\\\n\", \"  File \\\\\"/app/./materializationengine/blueprints/client/api2.py\\\\\", line 758, in get\\\\n    ann_table = ann_md.pop(\\\\\"annotation_table\\\\\", None)\\\\n\", \"AttributeError: \\'NoneType\\' object has no attribute \\'pop\\'\\\\n\"]}\\n'"
}

Expected output

['nucleus_detection_v0',
 'synapses_pni_2',
 'nucleus_neuron_svm',
 'proofreading_status_public_release',
 'func_unit_em_match_release',
 'allen_soma_ei_class_model_v1',
 'allen_visp_column_soma_coarse_types_v1',
 'allen_soma_coarse_cell_class_model_v1']

Your system

schlegelp commented 8 months ago

Thanks for the report, Jakob!

I hadn't looked at this dataset in a while but it seems the datastacks were updated: for the cortex65 it used to be just minnie65_public_v117 but there are now multiple versions, with minnie65_public_v661 being the latest.

>>> import navis.interfaces.microns as mi
>>> client = mi.get_cave_client(datastack='cortex65')
>>> sorted(client.info.get_datastacks())
['fanc_production_mar2021',
 'fanc_sandbox',
 'flywire_fafb_production',
 'flywire_fafb_public',
 'flywire_fafb_sandbox',
 'minnie35_public_v0',
 'minnie65_public',
 'minnie65_public_v117',
 'minnie65_public_v343',
 'minnie65_public_v661',
 'minnie65_sandbox',
 'mrgd',
 'pinky_sandbox']

I will have to rework the functions to always use the latest but in the meantime, here is a workaround:

>>> import navis.interfaces.microns as mi
>>> client = mi.get_cave_client(datastack='minnie65_public_v661')
>>> client.materialize.get_tables()
['baylor_gnn_cell_type_fine_model_v2',
 'nucleus_alternative_points',
 'connectivity_groups_v507',
 'proofreading_status_public_release',
 'allen_column_mtypes_v1',
 'allen_v1_column_types_slanted_ref',
 'aibs_column_nonneuronal_ref',
 'nucleus_ref_neuron_svm',
 'aibs_soma_nuc_exc_mtype_preds_v117',
 'baylor_log_reg_cell_type_coarse_v1',
 'apl_functional_coreg_forward_v5',
 'nucleus_detection_v0',
 'aibs_soma_nuc_metamodel_preds_v117']
wanqingy commented 1 month ago

Hi @schlegelp , a related question: I was not able to fetch a neuron from v661

`rid = 864691135162983725

n = mi.fetch_neurons( rid, with_synapses=True, datastack='minnie65_public_v661' ) n`

the message I got:

864691135162983725 generated an exception: GrapheneUnshardedMeshSource.get() got an unexpected keyword argument 'progress'

bdpedigo commented 1 month ago

@wanqingy what version of cloudvolume are you on?

wanqingy commented 1 month ago

11.0.1

schlegelp commented 1 month ago

Hi @wanqingy. Your example works for me with navis 1.9.0 and cloud-volume 11.0.1 or 11.0.2. What version of navis are you on?

I should add though that it's currently failing if with_synapses=True but with a different error and that seems to be on the CAVE end.

@bdpedigo: for that datastack, client.materialize.synapse_table is pointing to "synapses_pni_2" which doesn't seem to exist for me. Consequently, client.materialize.synapse_query() is failing. Is that a configuration issue?

wanqingy commented 1 month ago

@schlegelp hi! Upgrading from navia 1.7.0 -> 1.9.0 solved the issue if with_synapses=False

@bdpedigo added an option to set materialization version in CAVEclient, so I normally just use 

version = 661
datastack = 'minnie65_public'
client = CAVEclient(datastack,version=version)
bdpedigo commented 1 month ago

my understanding is that the 'minnie65_public_v661' syntax is no longer preferred; if I remember correctly, it is hitting a completely different version of the server but @fcollman correct me if I'm wrong. regardless I think providing a passthrough for setting a version of the client like Wan-Qing showed would be more sustainable going forward, and would let people specify arbitrary datastack versions (although, for just getting the mesh from a root ID, this shouldn't matter, but for getting synapses it will).

schlegelp commented 1 month ago

OK, I see. So minnie65_public is the preferred datastack and on top of that, the root ID you're querying is not current?

Our package for the FlyWire dataset - fafbseg - has a bunch of utility functions and guardrails to e.g. match root IDs to materializations. We don't provide the same functionality in the built-in microns interface at the moment but minimally we can provide the option to specify a materialization version and possibly some clever way to find a matching version by default.

schlegelp commented 1 month ago

I just made a new release. If you update navis to 1.9.1 this should just work:

>>> import navis.interfaces.microns as mi
>>> n = mi.fetch_neurons(864691135162983725, with_synapses=True, datastack='cortex65', materialization="auto")
>>> n[0].connectors.head()
          id        x       y        z type  size
0  299725236  1036344  786632   774680  pre  8704
1  332436677  1114904  828216   998200  pre  9088
2  197429586   799496  855720  1013280  pre  4288
3  292971356  1023800  805720   758760  pre  3952
4  330339468  1123520  804848   864360  pre  4612
  1. datastack='cortex65' now correctly points to minnie65_public
  2. mi.fetch_neurons() now also accepts a materialization parameter that lets you specify a version. It defaults to "auto" in which case it just tries to find the latest materialization in which your root ID(s) exist.
wanqingy commented 1 month ago

@schlegelp Thank you!