custom bulker prompt

[jpsmith5]

It would be helpful to have some sort of indicator that I have activated a bulker manifest and am currently residing in that space. Something along the lines of a python virtual environment would do the trick. e.g.

$ bulker activate databio/peppro
Bulker config: /mnt/storage/bulker_cfg.yaml
Activating bulker crate: databio/peppro

(databio/peppro) $ 

[nsheff]

I've got this sort of working. I can do it on strict mode, or if you use the -e and bash eval trick.

I can't get it to work completely on normal mode, though -- because the user's .bashrc gets run, which typically overrides the PS1 variable. So, I set the variable, but then the user overwrites it anyway. Not sure there's much I can do about that... at least it works in strict mode.

For the ones where it works, I made it so you can set your PS1 variable with a shell_prompt attribute in the bulker config; you can use \b to refer to the bulker registry path.

Here's the default one:

shell_prompt: "\[\033[01;93m\]\b▣\[\033[00m\] \[\033[01;34m\]\w\[\033[00m\]\$ "

[jpsmith5]

Oh great. So, the issue is just that, the user .bashrc is always going to be sourced after bulker does its thing. So there's no scenario in which after sourcing bashrc you can then update the PS1?

[nsheff]

exactly.

[nsheff]

should now work on dev! even in normal mode now.

[jpsmith5]

Couple errors I see popping up (on Rivanna):

1. PS1 formatting is off; maybe as a reflection of my .bashrc settings? BEFORE: [07:26 AM jps3dp@udc-ba35-36 atac_seq]$ AFTER: \[\033[01;93m\]\bulker/coreutils▣\[\033[00m\] \[\033[01;34m\]\w\[\033[00m\]\$

2. After activating crate; singularity pull fails with following error: Error for command "pull": unknown shorthand flag: 'n' in -n

Going to test the dev changes on Windows to see if that resolves issues I was having there. I can get docker working on WSL. So I can run Ubuntu on Windows. Within Ubuntu, I can successfully docker run. I can install bulker no issue, BUT, when I load it fails. I can't figure out where the connection is disrupted. Maybe PATH changes are affecting it? Will test with these changes just to see if my results are any different.

[jpsmith5]

Does the Docker daemon messaging point you to anything you're aware of?

I've tried looking at ways of linking the IP using DOCKER_HOST envvars and such but no luck. I can clearly run docker from Ubuntu on Windows, but within bulker, no go.

$ bulker load bulker/demo
Bulker config: /home/jps3dp/tools/bulker_config.yaml
Got URL: http://hub.bulker.io/bulker/demo.yaml
Loading manifest: 'bulker/demo:default'. Activate with 'bulker activate bulker/demo:default'.
Commands available: cowsay, fortune
$ bulker activate bulker/demo
Bulker config:/home/jps3dp/tools/bulker_config.yaml
Activating bulker crate: bulker/demo
$which cowsay
/home/jps3dp/bulker_crates/bulker/demo/default/cowsay
$cowsay
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.
$docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.

[nsheff]

are you using the lab bulker config or your own? echo $BULKERCFG

[nsheff]

PS1 formatting is off; maybe as a reflection of my .bashrc settings?

No, it's a bulker issue. put this into your bulker config:

  shell_prompt: \[\033[01;93m\]\b▣\[\033[00m\] \[\033[01;34m\]\w\[\033[00m\]\$ 

or take out the shell_prompt

[nsheff]

After activating crate; singularity pull fails with following error:

Ok I solved this with the latest version. it's a singularity version compatibility issue.

[nsheff]

Does the Docker daemon messaging point you to anything you're aware of?

I can'ty explain why it

try running docker command directly as it is in the shell script:

cat ` which cowsay`

[nsheff]

No, it's a bulker issue. put this into your bulker config:

Actually, still working on this... it's a problem with load changing the prompt...

[nsheff]

it's adding new slashes when it reads in any string with backslashes:

https://github.com/databio/yacman/issues/32

So for now the solution is to just remove the custom 'shell_prompt".

[jpsmith5]

Rivanna:

I used both the lab version, and created a fresh one following the dev branch install of bulker. Had to manually replace the shell_prompt in config file and it worked correctly.

Still getting same singularity error after updating dev and reinstalling?

Ubuntu on Windows: Full command leads to: C:\Program Files\Docker\Docker\resources\bin\docker.exe: invalid reference format. Reducing command to just:

docker run --rm --init -i nsheff/cowsay cowsay "$@" hello world
Unable to find image 'nsheff/cowsay:latest' locally 
latest: Pulling from nsheff/cowsay
c64513b74145: Already exists
01b8b12bad90: Already exists
c5d85cf7a05f: Already exists
b6b268720157: Already exists
e12192999ff1: Already exists
834a54f7272b: Pull complete
Digest: sha256:14fa1f533678750afd09536872e068e732ae4f735c52473450495d5af760c2e3
Status: Downloaded newer image for nsheff/cowsay:latest
 _________________
<   hello world   >
 -----------------
    \   ^__^
     \  (oo)\_______
         (__)\       )\/\
              ||----w |
              ||     ||

worked...

[jpsmith5]

--volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" \

At least one issue is this volume it tries to mount. Leads to "C:\Program Files\Docker\Docker\resources\bin\docker.exe: invalid reference format."

Same for this location: --volume "/mnt/c/Program\\ Files/Docker/:/mnt/c/Program\\ Files/Docker/" \

Leads to "C:\Program Files\Docker\Docker\resources\bin\docker.exe: Error response from daemon: /mnt/c/Program\ Files/Docker/%!(EXTRA string=is not a valid Windows path)."

Remove both volume's and the command executes like so:

docker run --rm --init \
-i \
--user=$(id -u):$(id -g) \
--network="host" \
--env "DISPLAY" \
--env "CODE" \
--env "CODEBASE" \
--env "PROCESSED" \
--env "DATA" \
--env "SRAFQ" \
--env "SRABAM" \
--env "DIVCFG" \
--env "REFGENIE" \
--env "PICARD" \
--volume "$HOME:$HOME" \
--volume="/etc/group:/etc/group:ro" \
--volume="/etc/passwd:/etc/passwd:ro" \
--volume="/etc/shadow:/etc/shadow:ro" \
--volume="/etc/sudoers.d:/etc/sudoers.d:ro" \
--workdir="`pwd`" \
nsheff/cowsay cowsay "$@" hello there

[nsheff]

you can remove those lines from the docker templates that ship with bulker. in your templates folder will be the version of this file:

https://github.com/databio/bulker/blob/master/bulker/templates/docker_executable.jinja2

that should at least correct it for that setup.

the others are coming from your bulker config, in the envvars section.

[jpsmith5]

Gotcha. Removed them from the executable. Still can't run cowsay at the command line following activation. Can only copy the full command and run it that way, which is just running docker manually yeah?

Process:

$ bulker activate bulker/demo
Bulker config: /home/jps3dp/tools/bulker_config.yaml
Activating bulker crate: bulker/demo 

$ cowsay Hey there!
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.

$docker run --rm --init \
-i \
--user=$(id -u):$(id -g) \
--network="host" \
--env "DISPLAY" \
--env "CODE" \
--env "CODEBASE" \
--env "PROCESSED" \
--env "DATA" \
--env "SRAFQ" \
--env "SRABAM" \
--env "DIVCFG" \
--env "REFGENIE" \
--env "PICARD" \
--volume "$HOME:$HOME" \
--volume="/etc/group:/etc/group:ro" \
--volume="/etc/passwd:/etc/passwd:ro" \
--volume="/etc/shadow:/etc/shadow:ro" \
--volume="/etc/sudoers.d:/etc/sudoers.d:ro" \
--workdir="`pwd`" \
nsheff/cowsay cowsay "$@" hello there
_____________ 
< hello there >
------------- 
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\ 
                ||----w |
                ||     ||

[jpsmith5]

Maybe just solved this...Found the proper DOCKER_HOST var setting.

export DOCKER_HOST=localhost:2375

Testing now to confirm.

[jpsmith5]

I think that was it. The tmp volume can also be included again and still works. I have this process written up in a markdown document. From no docker and no WSL to bulker working. Is there somewhere I should drop this for the lab? For any other windows people? Not sure if it would be helpful to other people.

[jpsmith5]

Well, as soon as I went to a heavier crate, I hit a snag:

$ bulker load databio/pepatac
Importing crate 'bulker/alpine:default' from '/home/jps3dp/bulker_crates/bulker/alpine/default'.
Importing crate 'bulker/coreutils:default' from '/home/jps3dp/bulker_crates/bulker/coreutils/default'.
Populating host commands
Traceback (most recent call last): 
    File "/home/jps3dp/.local/bin/bulker", line 11, in <module>
        load_entry_point('bulker==0.4.0', 'console_scripts', 'bulker')(
    File "/home/jps3dp/.local/lib/python3.6/site-packages/bulker/bulker.py", line 843, in main
        force=args.force)
    File "/home/jps3dp/.local/lib/python3.6/site-packages/bulker/bulker.py", line 441, in bulker_load
        os.symlink(local_exe, path)
    FileExistsError: [Errno 17] File exists: '/usr/bin/python' -> '/home/jps3dp/bulker_crates/databio/pepatac/default/python'

So, not quite there yet.

[jpsmith5]

Something about importing host commands throws that File exists error.

[jpsmith5]

Even just trying to load bulker/alpine it throws same error for id, which coincidentally is the first host command to try to include.

[jpsmith5]

Hmm. It's really confusing. I can load and activate bulker/alpinebase, which imports alpine. But id won't function, just hangs at command line. Whereas things like awk and grep work as expected...

[nsheff]

is there a working id command outside of bulker? it should just be a symlink to that.

[jpsmith5]

Yes it does.

[04:22 PM jps3dp@DESKTOP-5B6P5VS ~]$ id
uid=1000(jps3dp) gid=1000(jps3dp) groups=1000(jps3dp),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),114(netdev),115(docker)

And then.

$ bulker load bulker/alpine
Bulker config: /home/jps3dp/tools/bulker_config.yaml
Populating host commands
Traceback (most recent call last):
  File "/home/jps3dp/.local/bin/bulker", line 11, in <module>
    load_entry_point('bulker==0.4.0', 'console_scripts', 'bulker')() 
  File "/home/jps3dp/.local/lib/python3.6/site-packages/bulker/bulker.py", line 843, in main
    force=args.force)
  File "/home/jps3dp/.local/lib/python3.6/site-packages/bulker/bulker.py", line 441, in bulker_load
    os.symlink(local_exe, path)
FileExistsError: [Errno 17] File exists: '/usr/bin/id' -> '/home/jps3dp/bulker_crates/bulker/alpine/default/id'

[jpsmith5]

[04:35 PM jps3dp@DESKTOP-5B6P5VS ~]$ which id
/usr/bin/id

[nsheff]

that just looks like the folder is not getting correctly cleared out...

nsheff@zither:~$ touch test
nsheff@zither:~$ touch test2
nsheff@zither:~$ ln -s test test2
ln: failed to create symbolic link 'test2': File exists
nsheff@zither:~$ rm test2
nsheff@zither:~$ ln -s test test2

[nsheff]

I can reproduce this if I manually remove the entry fromt he config file, but don't remove the files in the folder itself:

bulker load bulker/alpine
Populating host commands
Traceback (most recent call last):
  File "/home/nsheff/.local/bin/bulker", line 10, in <module>
    sys.exit(main())
  File "/home/nsheff/.local/lib/python3.7/site-packages/bulker/bulker.py", line 843, in main
    force=args.force)
  File "/home/nsheff/.local/lib/python3.7/site-packages/bulker/bulker.py", line 441, in bulker_load
    os.symlink(local_exe, path)
FileExistsError: [Errno 17] File exists: '/usr/bin/id' -> '/home/nsheff/bulker_crates/bulker/alpine/default/id'

[jpsmith5]

Sorry, not sure I follow for my side. Is this suggesting something I am doing wrong or need to change?

[nsheff]

I'm not sure how you created that error but it's happening at load, not after activation... something is not working right. Look at:

ls /home/jps3dp/bulker_crates/databio/pepatac/default/

if you're loading it, it should be empty. if there's stuff in there, but not in your config, you can't reload it...

you can delete them all

rm /home/jps3dp/bulker_crates/databio/pepatac/default/*

[nsheff]

i think it may have to do with partially finished stuff from things that didn't complete all the way on earlier command executions

[jpsmith5]

Hah! You were 100% correct. Just to cover all my bases, removed everything currently present in the bulker_crates/. But to additionally confirm the culprit, I noticed all the links/files were broken in the alpine/default/ location, which is where it was having trouble loading. Can now successfully load , activate, and call id for example.

Now to resolve issues outside of host_commands...Still have problems running anything that is using docker.

[nsheff]

Ok this issue is about a year old and I'm not sure where we stand... I do know that bulker now does change the prompt for you.

However, it does not allow one to customize that. I'm going to close this in favor of #50, which is that issue.

@jpsmith5 please re-open this issue if there was anything else unsolved here.

[nsheff]

FYI custom prompts are now documented and fully working.