"basename: missing operand" at end of pycbc_submit_dax (for inference workflows)

[dbkeitel]

@duncan-brown (as you seem to be the script's main author): Submitting pycbc_inference workflows at CIT, I'm always seeing as the last output something like:

Querying Pegasus database for workflow stored in /local/david.keitel/pycbc-tmp.FZX36jC16X/work
This may take up to 120 seconds. Please wait....................................... Done.
basename: missing operand
Try 'basename --help' for more information.

The workflow is then intact and running, but from having a look at the pycbc_submit_dax script it seems there should have been some more helpful output (dashboard link) coming where the basename error message is instead thrown. Not sure if this is reproducible on other machines, sorry.

I've checked that the DASHBOARD_GLOB and DASHBOARD_PATH variables are actually empty for me here, and then it seems that the (to me counterintuitive) logic of bash for the if [ -e ${DASHBOARD_PATH} ] statement on L605 is that it evaluates to true, so that it will indeed try to call basename with empty (missing) operand.

If I comment out that if-block, I get the otherwise missing output:

Workflow submission completed successfully.

The Pegasus dashboard URL for this workflow is:
  https://ldas-grid.ligo.caltech.edu/pegasus/u/david.keitel/r/15/w?wf_uuid=df3803e5-4654-4b6f-8bf7-aa3bd9d29ed2

Note that it make take a while for the dashboard entry to appear while the workflow
is parsed by the dashboard. The delay can be on the order of one hour for very large
workflows.

Though I don't understand the path structures well enough to know if this is now a valid URL or not. (It's throwing an "Unknown Error" for my test run as of now, but since there's the comment about delays, I'll check again later and update if it works or not.)

Bottom line: not sure if it's just that the one if-statement needs an extra clause for an empty string, or if something is going wrong earlier on our setup. Input appreciated.

[dbkeitel]

@spxiwh Do you know if this dashboard feature is expected to work on CIT, i.e. whether the suggested fix should be "make this work" or just "make it fail more elegantly"?

[spxiwh]

@dbkeitel Well the dashboard feature is not running reliably anywhere (this is something Duncan was responsible for that we haven't really managed to replicate without him). However, normally the problem is that you get a link that is not going to be useful, rather than a failure. I'm not sure why this is happening in your case, but there does seem to be an issue with the bash logic, as you mention.

[cdcapano]

I've been getting this issue too. Assigning @duncan-brown and @cmbiwer to it.

[duncan-brown]

@cdcapano I thought that I fixed this. Is this with current master?