search

DigitalSlideArchive / HistomicsTK

A Python toolkit for pathology image analysis algorithms.
https://digitalslidearchive.github.io/HistomicsTK/
Apache License 2.0
394 stars 117 forks source link

'odict_keys' object has no attribute 'reverse' #379

Closed hsinewu closed 5 years ago

hsinewu commented 7 years ago

Ubuntu 16.04, reading doc

I try to install with docker but when it starts, localhost:8080 shows connection was reset

So I try to pull updates, wonder would it fix the problem, but then I got the following error when I try to run deploy_docker stop or rm.

$ python deploy_docker.py stop # or rm
Traceback (most recent call last):
  File "deploy_docker.py", line 869, in <module>
    containers_stop(remove=args.command in ('rm', 'remove'))
  File "deploy_docker.py", line 418, in containers_stop
    keys.reverse()
AttributeError: 'odict_keys' object has no attribute 'reverse'

It seems like the container is still occupying 8080, and since stop doesn't work I can't shut it down.

btw not sure if it matters, but seems like I didn't stop the container before pulling updates.

manthey commented 7 years ago

It sounds like a Python 3 issue. I don't think we've tested the deploy_docker.py script on Python 3 (only on Python 2). I'll see what needs to be done to make it Python 3 compatible.

In the mean time, you can stop and remove the containers manually. Type docker ps -a to list the containers. docker rm -f <container id> <container id ...>. You can probably also remove them by name: docker rm -f histomicstk_histomicstk histomicstk_girder_worker histomicstk_mongodb histomicstk_rmq.

hsinewu commented 7 years ago

Thanks, I can remove the containers with that command, now I have another problem.

Running start shows following message.

$ python deploy_docker.py start
Creating rabbitmq:management - histomicstk_rmq
Starting rabbitmq:management - histomicstk_rmq
Creating mongo:latest - histomicstk_mongodb
Starting mongo:latest - histomicstk_mongodb
Creating dsarchive/girder_worker:v0.1.3 - histomicstk_girder_worker
Starting dsarchive/girder_worker:v0.1.3 - histomicstk_girder_worker
Creating dsarchive/histomicstk_main:v0.1.3 - histomicstk_histomicstk
Starting dsarchive/histomicstk_main:v0.1.3 - histomicstk_histomicstk
Waiting for Girder to report version: ..Traceback (most recent call last):
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/docker/api/client.py", line 220, in _raise_for_status
    response.raise_for_status()
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/requests/models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localunixsocket/v1.24/containers/0b26136dfdbd720ed2b4b6f141900a50a2e5458301cf0a050425340f2c1ac34f/exec

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "deploy_docker.py", line 871, in <module>
    containers_start(**vars(args))
  File "deploy_docker.py", line 165, in containers_start
    containers_provision(**kwargs)
  File "deploy_docker.py", line 108, in containers_provision
    wait_for_girder(client, ctn)
  File "deploy_docker.py", line 726, in wait_for_girder
    container=ctn.get('Id'), cmd=exec_command, tty=True)
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/docker/utils/decorators.py", line 34, in wrapper
    return f(self, *args, **kwargs)
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/docker/api/exec_api.py", line 71, in exec_create
    return self._result(res, True)
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/docker/api/client.py", line 226, in _result
    self._raise_for_status(response)
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/docker/api/client.py", line 222, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 409 Client Error: Conflict ("Container 0b26136dfdbd720ed2b4b6f141900a50a2e5458301cf0a050425340f2c1ac34f is restarting, wait until the container is running")

Then, run it again will show

$ python deploy_docker.py start
Starting dsarchive/girder_worker:v0.1.3 - histomicstk_girder_worker
Starting dsarchive/histomicstk_main:v0.1.3 - histomicstk_histomicstk

Seems like for some reason the execution of container=ctn.get('Id'), cmd=exec_command, tty=True) in wait_for_girder(client, ctn) is too early?

I guess that's probably not a big problem and should still work? But it turns out that opening localhost:8080 only shows connection was reset (instead of unable to connect).

How do you think? Thanks a lot.

hsinewu commented 7 years ago

Try to update docker to docker-ce (was running old version from default repository).

Output of start are pretty much the same(still getting 409 conflict), but now showing unable to connect from localhost:8080. Also 8080 was not listed when I look up listening ports anymore.

Output of docker ps -a for the time being.

$ docker ps -a
CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS                          PORTS                                                 NAMES
5fea74ba06f3        dsarchive/histomicstk_main:v0.1.3   "/bin/sh -c 'sudo ..."   9 minutes ago       Restarting (2) 19 seconds ago                                                         histomicstk_histomicstk
795673ec2128        dsarchive/girder_worker:v0.1.3      "/bin/sh -c 'sudo ..."   9 minutes ago       Restarting (2) 22 seconds ago                                                         histomicstk_girder_worker
b1ca4c95b943        mongo:latest                        "docker-entrypoint..."   9 minutes ago       Up 9 minutes                    27017/tcp                                             histomicstk_mongodb
2f23a5fd4fff        rabbitmq:management                 "docker-entrypoint..."   9 minutes ago       Up 9 minutes                    4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   histomicstk_rmq
manthey commented 7 years ago

Can you try deploy_docker.py rm to remove the existing containers, then deploy_docker.py start --latest to try to use the latest ones? Without the --latest flag, we install from a pinned version, and I suspect that PR #363 made it so that the deploy_docker script no longer works with the pinned version. If that works, then I'll update which versions of our docker files we are pinned to.

hsinewu commented 7 years ago

Try deploy_docker.py start --latest and doesn't make any difference, except for the message showing that it's pulling dsarchive/girder_worker:latest and dsarchive/histomicstk_main:latest.

If you have a tested run, maybe I can checkout that version and see if it works?

manthey commented 7 years ago

Can you try again using the update-pinned-docker-versions branch? It should fix some Python 3 issues, too.

hsinewu commented 7 years ago

:heart: python 3 issue fixed (tested start/ rm) :broken_heart: containers still not functioning

Running start

$ python ansible/deploy_docker.py start
Creating rabbitmq:management - histomicstk_rmq
Starting rabbitmq:management - histomicstk_rmq
Creating mongo:latest - histomicstk_mongodb
Starting mongo:latest - histomicstk_mongodb
Pulling dsarchive/girder_worker:v0.1.5
Creating dsarchive/girder_worker:v0.1.5 - histomicstk_girder_worker
Starting dsarchive/girder_worker:v0.1.5 - histomicstk_girder_worker
Pulling dsarchive/histomicstk_main:v0.1.5
Creating dsarchive/histomicstk_main:v0.1.5 - histomicstk_histomicstk
Starting dsarchive/histomicstk_main:v0.1.5 - histomicstk_histomicstk
Waiting for Girder to report version: Traceback (most recent call last):
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/docker/api/client.py", line 220, in _raise_for_status
    response.raise_for_status()
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/requests/models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localunixsocket/v1.30/containers/37a4200a3d610675085e960d818f8247a2a41c6384b37801a2f18884cf4a3554/exec

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "ansible/deploy_docker.py", line 891, in <module>
    containers_start(**vars(args))
  File "ansible/deploy_docker.py", line 165, in containers_start
    containers_provision(**kwargs)
  File "ansible/deploy_docker.py", line 108, in containers_provision
    wait_for_girder(client, ctn)
  File "ansible/deploy_docker.py", line 746, in wait_for_girder
    container=ctn.get('Id'), cmd=exec_command, tty=True)
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/docker/utils/decorators.py", line 34, in wrapper
    return f(self, *args, **kwargs)
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/docker/api/exec_api.py", line 71, in exec_create
    return self._result(res, True)
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/docker/api/client.py", line 226, in _result
    self._raise_for_status(response)
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/docker/api/client.py", line 222, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/home/hsinewu/anaconda3/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 409 Client Error: Conflict ("Container 37a4200a3d610675085e960d818f8247a2a41c6384b37801a2f18884cf4a3554 is restarting, wait until the container is running")

start again

$ python ansible/deploy_docker.py start
Starting dsarchive/girder_worker:v0.1.5 - histomicstk_girder_worker
Starting dsarchive/histomicstk_main:v0.1.5 - histomicstk_histomicstk

8080 port is NOT listening docker ps -a

$ docker ps -a
CONTAINER ID        IMAGE                               COMMAND                  CREATED             STATUS                          PORTS                                                 NAMES
0cc23e9a9b6e        dsarchive/histomicstk_main:v0.1.5   "/bin/sh -c 'sudo ..."   7 minutes ago       Restarting (1) 33 seconds ago                                                         histomicstk_histomicstk
5e0a0aa19331        dsarchive/girder_worker:v0.1.5      "/bin/sh -c 'sudo ..."   7 minutes ago       Restarting (1) 30 seconds ago                                                         histomicstk_girder_worker
a144c54d1b4d        mongo:latest                        "docker-entrypoint..."   7 minutes ago       Up 7 minutes                    27017/tcp                                             histomicstk_mongodb
5bfb8a8c91da        rabbitmq:management                 "docker-entrypoint..."   7 minutes ago       Up 7 minutes                    4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   histomicstk_rmq

Also tried vagrant and deploy_local.sh, these are what I got

~/pj/HistomicsTK/ansible$ ./deploy_local.sh 
- girder.girder is already installed, skipping.
SUDO password: 
ERROR! no action detected in task

The error appears to have been in '/home/hsinewu/pj/HistomicsTK/ansible/roles/mongodb/tasks/main.yml': line 55, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- name: Run mongo daemon
  ^ here
$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Checking if box 'bento/ubuntu-16.04' is up to date...
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 (guest) => 2209 (host) (adapter 1)
    default: 8080 (guest) => 8009 (host) (adapter 1)
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
    default: The guest additions on this VM do not match the installed version of
    default: VirtualBox! In most cases this is fine, but in rare cases it can
    default: prevent things such as shared folders from working properly. If you see
    default: shared folder errors, please make sure the guest additions within the
    default: virtual machine match the version of VirtualBox you have installed on
    default: your host and reload your VM.
    default: 
    default: Guest Additions Version: 5.1.26
    default: VirtualBox Version: 5.0
==> default: Mounting shared folders...
    default: /vagrant => /home/hsinewu/pj/HistomicsTK
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.

Now 8009 port is listenning but get connection was reset from browser. Try to run tests but there is no /opt/histomicstk/build, only /opt/VBoxGuestAdditions-5.1.26. :flushed:

manthey commented 7 years ago

@hsinewu Can you try from the latest master branch? We've merged a bunch of fixes. I'd expect vagrant up to work without issue. I hope that when using Python 3, deploy_docker.py start --pull will also work.

hsinewu commented 7 years ago

I don't know why but vagrant doesn't work for me. (HEAD is 509f9bfd9dedff3d768a60e75990f88784268cb7) So I run netstat -tln and it shows the 8009 as expected.

hsinewu@hsinewu:~$ netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:8009            0.0.0.0:*               LISTEN   
// ...and more

But in the vm it only shows

vagrant@vagrant:~$ netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN

Is this reasonable? I was expecting port 8080 in the list. Because that's where localhost:8009 will be direct to. (Tried docker too but no)

manthey commented 7 years ago

What version of Vagrant do you have (and what host OS)? I notice a number of issues files against Vagrant >= 1.9.6. I just tried version Vagrant 2.0.0 on Windows 7, and it didn't work (it didn't print anything). Going back to my previous Vagrant version (1.8.6), worked.

hsinewu commented 7 years ago

Host: Ubuntu 16.04 Migrated from vagrant v1.8.1 to v1.8.6 (checked with vagrant version) Not worked. Don't see any difference.

I'm not sure how host OS can be relevant in this case, given that we are already using vm.

manthey commented 5 years ago

We've refactored many things since this issue was opened, including doing extensive testing on various versions of Python. Please open a new issue if there is a problem with Vagrant.

choosehappy commented 4 years ago

note to self: do not launch deploy_docker.sh as root, otherwise the above error messages will appear : )

using as a non-root, the same approach will work fine