Dockerized ice not responding/timeout; stops scraping info at same spot according to log

[jbz]

I'm trying to get the dockerized Ice running on OS X. I'm using the default settings from jbrouse's repo with of course my AWS key/keyid, bucket name and prefix in it. I can't get anything on the nginx port - nginx reports an upstream timeout to my browser (so I know I"m reaching nginx). Whenever I run docker-compose, ice starts scraping price information from region ap-south-1, and it stops producing output every time after (I should point out that my bucket is in us-west-1 and I set it as such).

ice_1    | 2018-05-15 16:15:48,578 [localhost-startStop-1] INFO  basic.BasicReservationService  - Setting RI price for ap-south-1 HEAVY x1.32xlarge.windows 1578185.0 0.0

I am using VirtualBox under docker-machine, and other docker workloads are working fine. I upped the VM's memory to 4G prior to running docker-machine, which doesn't seem to make a difference. I don't see any other error messages beyond the nginx timeout error. I tried upping the proxy_read-_timeout via an additional proxy_settings.conf for nginx, and it's obeying that larger timeout, but still timing out.

I've verified the AWS keypair works, and can read the billing reports and write to the ice reports prefix I assigned.

Any suggestions would be most gratefully appreciated.

[jbz]

UPDATE: Hm, I hadn't put the GRAIL_OPTS in to increase the server memory usage. Once I did that, the scraping process seems to have reached eu-west-1, which is further along. I'll see if this run ever starts answering the web proxy, and if it does, I'll close this issue. (For reference, the GRAIL_OPTS setting change can be found here.)

[nfonrose]

Ok, thanks for the update!

-- Nicolas Fonrose | Teevity | Founder +33.6.61.35.43.31 https://teevity.com - Cloud Costs Analytics built on NetflixOSS twitter - @nfonrose / @teevity

On Tue, May 15, 2018 at 10:43 PM, J.B. Zimmerman notifications@github.com wrote:

UPDATE: Hm, I hadn't put the GRAIL_OPTS in to increase the server memory usage. Once I did that, the scraping process seems to have reached eu-west-1, which is further along. I'll see if this run ever starts answering the web proxy, and if it does, I'll close this issue. (For reference, the GRAIL_OPTS setting change can be found here https://github.com/jonbrouse/docker-ice#outofmemory-exception.)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Teevity/ice/issues/285#issuecomment-389306934, or mute the thread https://github.com/notifications/unsubscribe-auth/AACUpYuTyQtYL0sxhZq4ldmbRWLLaxyZks5tyz4JgaJpZM4UANVb .

[jbz]

Second update: It got further this time before apparently stalling:

ice_1    | 2018-05-15 16:53:26,379 [localhost-startStop-1] INFO  basic.BasicReservationService  - Setting RI price for ap-northeast-2 HEAVY x1.32xlarge.windows 1586715.0 0.0

I note, however, that the container is still pulling in network I/O, which would seem to indicate it's still doing stuff, it just stopped logging. To wit:

And I've verified that AWS thinks its keypair was last used at 1708, despite the last log entry (the one above) being timestamped 1653. So I'm going to let it run, esp. since it doesn't appear to be anywhere near running out of resources. I just think it's taking its time and is having trouble logging, and I'll see if it responds to net traffic once the net i/o drops.

[jbz]

...nope. Eventually, the net I/O counters stopped moving after about an hour, but nginx still returns an upstream timeout, and no additional log messages have shown up from ice_1. :-( The container is still using less than 700 MiB, or < 18% of its available RAM, and approx 0.12% CPU. It's just not answering nginx. Or logging.

Any suggestions? Is there any way to limit it to just scraping the data from one AWS region? And should it be answering network requests while scraping, or not?

[jbz]

When I ctrl-c the docker-compose command, additionally, I always get something quite like:

^CGracefully stopping... (press Ctrl+C again to force)
Stopping docker-ice_nginx_1   ... done
Stopping docker-ice_ice_1     ...

ERROR: for docker-ice_ice_1  HTTPSConnectionPool(host='192.168.99.102', port=2376): Read timed out. (read timeout=190)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 180).

[jbz]

(add'l info - also can't docker attach, docker exec, or docker inspect the ice container. Just hangs.)

[nfonrose]

Sorry I can't help. I have no experience running Ice on Docker on a local machines. Have you tried running jbrouse Docker Ice image on AWS, with the same ice.properties file ?

-- Nicolas Fonrose | Teevity | Founder +33.6.61.35.43.31 https://teevity.com - Cloud Costs Analytics built on NetflixOSS twitter - @nfonrose / @teevity

On Wed, May 16, 2018 at 12:33 AM, J.B. Zimmerman notifications@github.com wrote:

(add'l info - also can't docker attach, docker exec, or docker inspect the ice container. Just hangs.)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Teevity/ice/issues/285#issuecomment-389334858, or mute the thread https://github.com/notifications/unsubscribe-auth/AACUpXIsIllHFg48tkZb4AqzFAl5ag_Wks5ty1eygaJpZM4UANVb .