Get amf-checker installed and working

[agstephens]

Get amf-checker working. Stages are:

• Get Python 3.7 running on Linux
• Set up virtual environment for project
• Fork amf-checker to your github
• Clone your amf-checker
• Modify requirements.txt to non-specific versions
• pip install -r requirements.txt - to install all dependencies.
• pip install --editable . --no-deps - to install local package as a link so that any local edits are reflected when you run.

Can you now run command-line script: download-from-drive?

[agstephens]

Possible issues:

• all code need python3ising ??? - before anything can run

Does this work?

python -c "from amf_check_writer import amf_checker"

[gapintheclouds]

Making progress. Done the authorisation part for download-from-drive, but needs checking as it doesn't run directly in Python 3.8 (as expected).

Only change to your list: pip install --editable . --no-deps.

[agstephens]

@gapintheclouds great stuff. Have you got experience of migrating to Python 3? There are various tools to do it, I suspect that code is mainly ready - so much of the work will be in updating interfaces to modules/libraries.

[gapintheclouds]

@agstephens Not yet. I've just started looking into it and found some random guides. Do you have any recommendations?

[agstephens]

@gapintheclouds I have always done it manually. However, there are tools. E.g.: https://docs.python.org/3/library/2to3.html

It might be worth running through that tool and doing diffs between each original module and the version output by 2to3.

[gapintheclouds]

@agstephens Thanks. Making some progress with that.

[gapintheclouds]

Haven't actually made any python 2 to 3 changes yet. Getting the following error when running download-from-drive /tmp/spreadsheets:

Traceback (most recent call last): File "/home/hugo/python/checker-env/bin/download-from-drive", line 11, in <module> load_entry_point('amf-check-writer', 'console_scripts', 'download-from-drive')() File "/home/hugo/python/amf-check-writer/amf_check_writer/download_from_drive.py", line 220, in main downloader = SheetDownloader(args.output_dir, secrets_file=args.secrets) File "/home/hugo/python/amf-check-writer/amf_check_writer/download_from_drive.py", line 94, in __init__ drive_service = service.DriveService(self.secrets_file) File "/home/hugo/python/checker-env/lib/python3.8/site-packages/pygdrive3/service.py", line 13, in __init__ client_secret_path = os.path.abspath(client_secret) File "/usr/lib/python3.8/posixpath.py", line 374, in abspath path = os.fspath(path) TypeError: expected str, bytes or os.PathLike object, not NoneType

It looks like DriveService is looking for a secrets_file, but it is set to None. I've tried:

drive_service = service.DriveService('/path/to/.credentials/drive.googleapis.com-python-quickstart.json')

This raises the following error:

oauth2client.clientsecrets.InvalidClientSecretsError: Invalid file format. See https://developers.google.com/api-client-library/python/guide/aaa_client_secrets Expected a JSON object with a single property for a "web" or "installed" application

Any thoughts?

[agstephens]

@gapintheclouds: It looks like the download script is trying to run and it is managing to try to connect to Google Drive. You will need to set up a local secrets.json file with the appropriate content in it.

Joe provided info about this authentication process, here:

https://github.com/ncasuk/amf-check-writer#authentication

It might need to be updated if the google documentation/process has changed.

[gapintheclouds]

@agstephens Thanks. I got that error after following the authentication step. It has created two secret files for both Drive and Sheets. Not sure what the next step is. We can troubleshoot later.

[gapintheclouds]

Got most of download-from-drive working. Still problems with the raw spreadsheet download part, so disabled that for now. Had to reduce the API requests quite a bit for some reason. Should be 100 requests per 100 seconds, but only getting between 20 and 30. Moving on to the next steps.

[gapintheclouds]

The code is running but create-cvs fails due to <> brackets (#14).

create-yaml is working, but shows warning messages for global-attributes-specific.tsv files: eg. WARNING: No match for '/tmp/spreadsheets/product-definitions/tsv/precipitation/global-attributes-specific.tsv'

[gapintheclouds]

@agstephens Things are just about running. I see that there are some test files in tests.py. Are they available so I can try it out?

[agstephens]

Great stuff @gapintheclouds: Please try this:

pytest amf_check_writer/tests.py

I suspect it will fail a number of tests. If so, it would be great to get them all working again. Thanks

[agstephens]

@gapintheclouds : I just tried to install from your fork with a completely new installation of python. I get a credentials error. Have you documented the workflow for how you got a valid credentials file that worked? Thanks

[gapintheclouds]

@agstephens, I'm pretty sure I followed the instructions here to get the credentials file: https://github.com/ncasuk/amf-check-writer#authentication and I think it just worked (except for the raw downloader bit). Does that work for you? I'll look into the raw downloader next.

[agstephens]

Thanks @gapintheclouds , I'll try it out again.

[agstephens]

Closing, it is working.