Missing file at ckanext.bigquery.google_cloud_credentials

[ccancellieri]

Hi, I'm eager to use your plugin! could be a great asset!

After having installed and configured your master branch i get the follow:

ERROR [src.api_tracker] Invalid URL 'None': No schema supplied. Perhaps you meant http://None?
 /etc/ckan/default/${MY_PROJECT}-270XXXXX.json
 None
 Error - <type 'exceptions.TypeError'>: invalid file: None
 URL: http://127.0.0.1:8080/dataset/hih-datasets-view-templates-csv/resource/5d8ac57c-fb6c-4476-a872-fc577132b5a2
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/weberror/errormiddleware.py', line 171 in __call__
   app_iter = self.application(environ, sr_checker)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
   resp = self.call_func(req, *args, **self.kwargs)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
   return self.func(req, *args, **kwargs)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__
   return request.get_response(self.app)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
   application, catch_exc_info=False)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
   app_iter = application(self.environ, start_response)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
   resp = self.call_func(req, *args, **self.kwargs)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
   return self.func(req, *args, **kwargs)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/fanstatic/injector.py', line 54 in __call__
   response = request.get_response(self.app)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
   application, catch_exc_info=False)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
   app_iter = application(self.environ, start_response)
 File '/usr/lib/ckan/default/src/ckan/ckan/config/middleware/pylons_app.py', line 264 in inner
   result = application(environ, start_response)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__
   return self.app(environ, start_response)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/beaker/middleware.py', line 156 in __call__
   return self.wrap_app(environ, session_start_response)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
   response = self.app(environ, start_response)
 File '/usr/lib/ckan/default/src/ckan/ckan/config/middleware/common_middleware.py', line 33 in __call__
   return self.app(environ, start_response)
 File '/usr/lib/ckan/default/src/ckan/ckan/config/middleware/common_middleware.py', line 59 in __call__
   return self.app(environ, start_response)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__
   response = self.dispatch(controller, environ, start_response)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch
   return controller(environ, start_response)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 242 in __call__
   res = WSGIController.__call__(self, environ, start_response)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 221 in __call__
   response = self._dispatch_call()
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
   response = self._inspect_call(func)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
   result = self._perform_call(func, args)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/controllers/core.py', line 60 in _perform_call
   return func(**args)
 File '/usr/lib/ckan/default/src/ckan/ckan/controllers/package.py', line 1147 in resource_read
   return render(template, extra_vars=vars)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 127 in render
   return cached_template(template_name, renderer)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/templating.py', line 249 in cached_template
   return render_func()
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 164 in render_template
   return render_jinja2(template_name, globs)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 96 in render_jinja2
   return template.render(**extra_vars)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/jinja2/environment.py', line 989 in render
   return self.environment.handle_exception(exc_info, True)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/jinja2/environment.py', line 754 in handle_exception
   reraise(exc_type, exc_value, tb)
 File '/usr/lib/ckan/default/src/ckan/ckanext/datastore/templates/package/resource_read.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/resource_read.html', line 3 in top-level template code
   {% set res = c.resource %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/base.html', line 3 in top-level template code
   {% set pkg = c.pkg_dict or pkg_dict %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/page.html', line 1 in top-level template code
   {% extends "base.html" %}
 File '/usr/lib/ckan/default/src/${MY_PREFIX}-ckanext-theme/ckanext/${CHANGE_ME}_theme/templates/base.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckanext-scheming_dcat/ckanext/scheming_dcat/templates/base.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckanext-googleanalytics/ckanext/googleanalytics/plugin/../templates/base.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckanext-geoview/ckanext/geoview/plugin/../templates/base.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckanext-scheming/ckanext/scheming/templates/base.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/plugin/../templates/base.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/base.html', line 101 in top-level template code
   {%- block page %}{% endblock -%}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/page.html', line 19 in block "page"
   {%- block content %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/page.html', line 22 in block "content"
   {% block main_content %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/page.html', line 53 in block "main_content"
   {% block pre_primary %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/resource_read.html', line 22 in block "pre_primary"
   {% block resource %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/resource_read.html', line 24 in block "resource"
   {% block resource_inner %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/resource_read.html', line 92 in block "resource_inner"
   {% block data_preview %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/resource_read.html', line 93 in block "data_preview"
   {% block resource_view %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/resource_read.html', line 106 in block "resource_view"
   {% block resource_view_content %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/resource_read.html', line 116 in block "resource_view_content"
   {% snippet 'package/snippets/resource_view.html',
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/jinja_extensions.py', line 268 in _call
   return base.render_snippet(args[0], **kwargs)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 86 in render_snippet
   output = render(template_name, extra_vars=kw)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 127 in render
   return cached_template(template_name, renderer)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/templating.py', line 249 in cached_template
   return render_func()
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 164 in render_template
   return render_jinja2(template_name, globs)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 96 in render_jinja2
   return template.render(**extra_vars)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/jinja2/environment.py', line 989 in render
   return self.environment.handle_exception(exc_info, True)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/jinja2/environment.py', line 754 in handle_exception
   reraise(exc_type, exc_value, tb)
 File '/usr/lib/ckan/default/src/${MY_PREFIX}-ckanext-theme/ckanext/${CHANGE_ME}_theme/templates/package/snippets/resource_view.html', line 1 in top-level template code
   {% ckan_extends %}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/snippets/resource_view.html', line 3 in top-level template code
   {% block resource_view %}
 File '/usr/lib/ckan/default/src/${MY_PREFIX}-ckanext-theme/ckanext/${CHANGE_ME}_theme/templates/package/snippets/resource_view.html', line 14 in block "resource_view"
   {{ super() }}
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/snippets/resource_view.html', line 24 in block "resource_view"
   {% snippet 'package/snippets/resource_view_filters.html', resource=resource %}
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/jinja_extensions.py', line 268 in _call
   return base.render_snippet(args[0], **kwargs)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 86 in render_snippet
   output = render(template_name, extra_vars=kw)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 127 in render
   return cached_template(template_name, renderer)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/pylons/templating.py', line 249 in cached_template
   return render_func()
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 164 in render_template
   return render_jinja2(template_name, globs)
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/base.py', line 96 in render_jinja2
   return template.render(**extra_vars)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/jinja2/environment.py', line 989 in render
   return self.environment.handle_exception(exc_info, True)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/jinja2/environment.py', line 754 in handle_exception
   reraise(exc_type, exc_value, tb)
 File '/usr/lib/ckan/default/src/ckan/ckan/templates/package/snippets/resource_view_filters.html', line 5 in top-level template code
   data-module-fields="{{ h.dump_json(h.resource_view_get_fields(resource)) }}"
 File '/usr/lib/ckan/default/src/ckan/ckan/lib/helpers.py', line 2309 in resource_view_get_fields
   result = logic.get_action('datastore_search')({}, data)
 File '/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py', line 467 in wrapped
   result = _action(context, data_dict, **kw)
 File '/usr/lib/ckan/default/src/ckan/ckanext/datastore/logic/action.py', line 466 in datastore_search
   result = backend.search(context, data_dict)
 File '/usr/lib/ckan/default/src/ckanext-datastore-bigquery/ckanext/bigquery/backend/bigquery.py', line 43 in search
   engine = self._get_engine()
 File '/usr/lib/ckan/default/src/ckanext-datastore-bigquery/ckanext/bigquery/backend/bigquery.py', line 30 in _get_engine
   self._engine = ckan2bq.Client(project, dataset, creds, read_only_creds)
 File '/usr/lib/ckan/default/src/ckanext-datastore-bigquery/src/ckan_to_bigquery.py', line 22 in __init__
   self.bqclient_readonly = bigquery.Client.from_service_account_json(read_only_creds)
 File '/usr/lib/ckan/default/lib/python2.7/site-packages/google/cloud/client.py', line 77 in from_service_account_json
   with io.open(json_credentials_path, "r", encoding="utf-8") as json_fi:
 TypeError: invalid file: None

 CGI Variables
 -------------
   AUTH_TYPE: 'cookie'
   CKAN_CURRENT_URL: '/dataset/hih-datasets-view-templates-csv/resource/5d8ac57c-fb6c-4476-a872-XXXXXXXX'
   CKAN_LANG: 'en'
   CKAN_LANG_IS_DEFAULT: True
   CONTEXT_DOCUMENT_ROOT: '/var/www/html'
   DOCUMENT_ROOT: '/var/www/html'
   GATEWAY_INTERFACE: 'CGI/1.1'
   HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
   HTTP_ACCEPT_ENCODING: 'gzip, deflate, br'
   HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.9,it;q=0.8'
...
 WSGI Variables
 --------------
   __no_cache__: True
   apache.version: (2, 4, 29)
   application: <fanstatic.publisher.Delegator object at 0x7fe04c26f950>
   beaker.cache: <beaker.cache.CacheManager object at 0x7fe04c450690>
   beaker.get_session: <bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7fe04c26f7d0>>
   beaker.session: {'_accessed_time': 1633119825.489249, '_creation_time': 1633104578.343546}
   ckan.app: 'pylons_app'
   fanstatic.needed: <fanstatic.core.NeededResources object at 0x7fe03ea53290>
   mod_wsgi.application_group: '${MY_PROJECT}-ckanx.{ZONE}.c.${MY_PROJECT}.internal:8080|/ckanx'
   mod_wsgi.callable_object: 'application'
   mod_wsgi.daemon_connects: '1'
   mod_wsgi.daemon_restarts: '0'
   mod_wsgi.daemon_start: '1633119825439052'
   mod_wsgi.enable_sendfile: '0'
   mod_wsgi.handler_script: ''
   mod_wsgi.ignore_activity: '0'
   mod_wsgi.listener_host: ''
   mod_wsgi.listener_port: '8080'
   mod_wsgi.path_info: '/dataset/hih-datasets-view-templates-csv/resource/5d8ac57c-fb6c-4476-a872-fc577132b5a2'
   mod_wsgi.process_group: 'ck

I've properly place my json file and granted right permissions.

Then configured production.ini with all the documented params plus: ckanext.bigquery.google_cloud_credentials = /etc/ckan/default/{MY_PROJECT}-270XXXXXX.json

Will give the above log.

SOLVED:

Configuring the following undocumented param the file is found:

ckanext.bigquery.google_cloud_credentials_read_only= /etc/ckan/default/{MY_PROJECT}-270XXXXXX.json

At the moment this is not the best, would you accept a quick contribution to add an option to use the machine service account? The machine has a service account configured so I don't need to manage this kind of secrets during the deploy. If so would you like to suggest or give indications on the way you prefer?

thanks C.

[ccancellieri]

Probably the above depicted behaviour is due to a wrong ckan.datastore.write_url setting, but my (OT) question is still valid.