dstat returns nothing for jobs submitted from cloud shell

[bertozzivill]

I run jobs with dsub both from the Google Cloud Shell and from the Google Cloud SDK installed on my desktop. When I submit a job from my desktop, dstat works as expected. But when I submit a job from the Cloud Shell, dstat returns nothing (when called from either Cloud Shell or my desktop SDK).

Below is a screenshot of my cloud console (in which I'd previously installed dsub using sudo pip install dsub), showing job submission and attempt to call dstat.

And below is my desktop command line (iTerm on OSX, with dsub installed via cloning this repo and running python setup.py --install), showing dstat working effectively on a job I'd previously submitted from desktop, but returning nothing on the job I ran from the cloud console:

Thanks in advance for your advice.

[mbookman]

Thank-you for the report @bertozzivill !

We will take a look.

[wnojopra]

Hi @bertozzivill , thanks for reporting. For a first look, could you please let us know what versions of dsub you have installed? On both your cloud console and your desktop, please run dstat --version and please report back. Thanks!

[bertozzivill]

Cloud shell is running 0.2.5, desktop is 0.2.2.

Oh, and I forgot to report the effects of running dstat on cloud console for a job that was submitted on desktop: it also returns nothing.

[wnojopra]

Hi @bertozzivill , I'm not able to reproduce the issue on my end, so doesn't seem to be a versioning issue.

I'm wondering if the user is different between the two consoles. On your dstat command, also add --user '*' . Please report if this changes anything for you.

Lets also check if we can get the job to show up in a list. It looks like you submitted your job on March 4, so lets check if it shows up in a list of jobs from the past 3 days. Run dstat --provider google-v2 --project {PROJECT_ID} --age '3d' --status '*' and see if that job shows up. Also try it with --user '*' added and see if that changes anything for you.

[bertozzivill]

Hi @wnojopra, thanks for your suggestions. See below for some experimentation with the --user tag (note that the last one is querying a job submitted from desktop, while the top ones are all the same job submitted from cloud shell). It seems like there are two forces at play here:

1. Job submitted from cloud shell get logged under the user "amelia-bertozzivilla" whereas those from my desktop fall under "bertozzivill";
2. The cloud shell doesn't seem to identify the user "amelia-bertozzivilla" as self, even though that's the user id under which it submits jobs.

How can I standardize my user id across platforms?

[wnojopra]

Hi @bertozzivill , thanks for answering back. On cloud shell, the user dsub and dstat uses is the google user. On desktop, it is whatever user that is logged in. To standardize your user id across platforms, I would suggest running dsub with the --user flag. From the docs: --user USER, -u USER User submitting the dsub job, defaults to the current OS user.

It is a bit concerning that on cloud shell, the default dstat command isn't fetching the job also submitted on cloud shell. dstat and dsub should both be using the same user id by default.

I do notice in your screenshot, there is a difference between amelia-bertozzivilla and amelia_bertozzivilla (hyphen vs underscore). Can you please run echo $USER on your cloud shell and please report back? If there is a hyphen or underscore in the name I will run some tests on my end to verify if there are any issues there.

Thanks!

[wnojopra]

I believe I have spotted an issue when there are non-alphanumeric non-hyphen characters in the username. I am working on a fix that will be available in the next release. In the meantime, you can use the --user flag as a workaround.

If you could also please still report back with the output of echo $USER is on the cloud shell, that would help confirm the issue. Thank you very much for all the help.

[bertozzivill]

Hi @wnojopra-- thanks for the tip about the --user flag, I'll use that to standardize my jobs moving forward. echo $USER on cloud shell returns amelia_bertozzivilla, consistent with the username displayed on the command line, but dstat does seem to instead be running jobs as amelia-bertozzivilla.

Thanks for your help!

[wnojopra]

Perfect, that confirms my suspicions that there was an issue in the conversion of underscores to hyphens (or more accurately, lack of conversion in dstat). I'll close this issue once the next release is available with the fix.

[wnojopra]

This is fixed now in release 0.2.6