Indexing ignoring new files for 01deg_jra55v140_iaf_KvJ09

[aidanheerdegen]

The automated indexing for 01deg_jra55v140_iaf_KvJ09 has not indexed any new files since 2021-05-11 19:41:58 despite there being files in /g/data/ik11//outputs/access-om2-01/01deg_jra55v140_iaf_KvJ09 that are not present in the DB.

It is occurring at the filtering step here:

https://github.com/COSIMA/cosima-cookbook/blob/master/cosima_cookbook/database.py#L702-L707

In the debugger

(Pdb) p len(files)
9800
(Pdb) tmp = {f for f, in session.query(NCFile.ncfile).with_parent(expt)}
(Pdb) len(files - tmp)
4410

So there are 4410 files that are not in the DB, but after the filtering step ncfiles is empty

(Pdb) b 709
Breakpoint 3 at /g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.04/lib/python3.8/site-packages/cosima_cookbook/database.py:709
(Pdb) c
> /g/data3/hh5/public/apps/miniconda3/envs/analysis3-21.04/lib/python3.8/site-packages/cosima_cookbook/database.py(709)build_index()
-> indexed += index_experiment(files, session, expt, client)
(Pdb) p len(files)
0
(Pdb) 

Current status: confused.

[aidanheerdegen]

@angus-g any ideas?

[AndyHoggANU]

Permissions??

[AndyHoggANU]

For what it's worth -- my temporary database for this experiment has the same problem! And permissions look fine. Maybe this is related to the change in the API which killed the update=True flag?

[aidanheerdegen]

Ok, weird, seems the query is returning a tuple now. How odd.

I'm struggling to see how that logic worked anyway.

Will work up a fix.

[aidanheerdegen]

Forget the tuple thing, red herring. As far as I can tell this never worked, but the test didn't insist that it did. Oops!