search

jupyter-server / enterprise_gateway

A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across distributed clusters such as Apache Spark, Kubernetes and others.
https://jupyter-enterprise-gateway.readthedocs.io/en/latest/
Other
617 stars 222 forks source link

KIP: Define pulled_images as set, not list #1125

Closed kevin-bates closed 2 years ago

kevin-bates commented 2 years ago

The changes in #1090 inadvertently changed the datatype of pulled_images from a set to a list in the Kernel Image Puller (KIP). This lead to the following issues for each image pulled (which caused KIP to pull all images on every iteration 3 times):

[I 2022-06-29 17:30:17,824 kernel_image_puller.t2] Pulled image 'elyra/kernel-r:kb0' in 1.127 secs.
[W 2022-06-29 17:30:17,824 kernel_image_puller.t2] Attempt 1 to pull image 'elyra/kernel-r:kb0' encountered exception - retrying.  Exception was: 'list' object has no attribute 'add'.
[I 2022-06-29 17:30:17,824 kernel_image_puller.t2] Pulling image 'elyra/kernel-r:kb0'...
[I 2022-06-29 17:30:18,959 kernel_image_puller.t2] Pulled image 'elyra/kernel-r:kb0' in 1.135 secs.
[W 2022-06-29 17:30:18,959 kernel_image_puller.t2] Attempt 2 to pull image 'elyra/kernel-r:kb0' encountered exception - retrying.  Exception was: 'list' object has no attribute 'add'.
[I 2022-06-29 17:30:18,959 kernel_image_puller.t2] Pulling image 'elyra/kernel-r:kb0'...
[I 2022-06-29 17:30:20,134 kernel_image_puller.t2] Pulled image 'elyra/kernel-r:kb0' in 1.174 secs.
[E 2022-06-29 17:30:20,134 kernel_image_puller.t2] Attempt 3 to pull image 'elyra/kernel-r:kb0' failed with exception: 'list' object has no attribute 'add'
[I 2022-06-29 17:30:20,134 kernel_image_puller.t2] waiting for new imag to pull

I also addressed the typo in the last log statement and changed it to DEBUG.

kevin-bates commented 2 years ago

With the changes, the KIP log shows similar to this...

[I 2022-06-29 18:37:03,010 kernel_image_puller.t2] Task received to pull image: elyra/kernel-tf-gpu-py:dev
[I 2022-06-29 18:37:03,010 kernel_image_puller.t2] Pulling image 'elyra/kernel-tf-gpu-py:dev'...
[I 2022-06-29 18:37:04,433 kernel_image_puller.t1] Pulled image 'elyra/kernel-py:dev' in 1.498 secs.
[D 2022-06-29 18:37:04,433 kernel_image_puller.t1] Waiting for new image to pull
[I 2022-06-29 18:37:04,433 kernel_image_puller.t1] Task received to pull image: elyra/kernel-spark-r:dev
[I 2022-06-29 18:37:04,433 kernel_image_puller.t1] Pulling image 'elyra/kernel-spark-r:dev'...
[I 2022-06-29 18:37:04,681 kernel_image_puller.t2] Pulled image 'elyra/kernel-tf-gpu-py:dev' in 1.670 secs.
[D 2022-06-29 18:37:04,681 kernel_image_puller.t2] Waiting for new image to pull
[I 2022-06-29 18:37:06,054 kernel_image_puller.t1] Pulled image 'elyra/kernel-spark-r:dev' in 1.620 secs.
[D 2022-06-29 18:37:06,054 kernel_image_puller.t1] Waiting for new image to pull
[I 2022-06-29 18:37:06,054 kernel_image_puller.MainThread] Sleeping 300 seconds to fetch image names...
rahul26goyal commented 2 years ago

thanks @kevin-bates for fixing it. Not sure how I missed this during my testing. Sorry about this.

kevin-bates commented 2 years ago

@rahul26goyal - no worries. Thanks for your review.