search

datopian / ckanext-iaea

Custom extension for IAEA CKAN
GNU Affero General Public License v3.0
0 stars 1 forks source link

[xs]: catch filter fields keyerror #31

Closed cynepton closed 2 years ago

cynepton commented 2 years ago

In some cases, this line:

fields = datapackage['resources'][0]['schema']['fields']

is called when the resource has not been pushed to datastore correctly, and that causes an internal server error for the user. If that happens, this fix catches the error and sets the fields variable to an empty array.

sagargg commented 2 years ago

@cynepton Did you check why datapackage doesn't have schemas? if it is pushed to datastore it is supposed to have filed schema. Maybe the issue is in dataexplorer extension. https://github.com/datopian/ckanext-dataexplorer-react/blob/18e1d0530ce7b9c9023d56b8f56b2862b333d337/ckanext/dataexplorer/plugin.py#L191-L193

you could confirm with the datastore_search API, check if it returns fields on whichever resource you encountered internal server error.

cynepton commented 2 years ago

@sagargg I was not able to replicate the error completely, so I'm not very sure of the actual cause of the issue Once the resource has been pushed to datastore, it has the schema, so that's fine But it seems datapackage['resources'][0]['schema']['fields'] is also being called on resources that weren't pushed to datastore and don't have the schema attribute yet. It think the cause may be that they failed to push to datastore at first, but since I couldn't replicate it, I don't know for sure. I just added the fix incase that happens to prevent the user from getting an internal server error

you could confirm with the datastore_search API, check if it returns fields on whichever resource you encountered internal server error.

I wasn't able to replicate the error the client had, so I can't test this again, but once the resource was pushed to datastore it had the schema. Before being uploaded, it didn't have the schema.

sagargg commented 2 years ago

Maybe this is the case when the resource has datastore_active= true attribute and datapusher failed to push data into datastore because of invalid datatype on records.

@cynepton okay. Let's merge this for now. I will validate the above hypothesis later.

LGTM