search

lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ā˜ļøšŸš€
http://lithops.cloud
Apache License 2.0
315 stars 103 forks source link

[Code Engine] Failed to run Lithops using Knative API of Code Engine #391

Closed zhanggbj closed 3 years ago

zhanggbj commented 3 years ago

My steps reproduce it

  1. Followed doc https://github.com/lithops-cloud/lithops/blob/master/config/compute/code_engine.md config file 
    
# cat config
cat config
lithops:
storage_bucket: lithops-serving
#storage_backend: ibm_cos
compute_backend: knative
#rabbitmq_monitor: <True/False>
#remote_invoker: <True/False>
#data_cleaner: <True/False>
#runtime : cactusone/pywren-betabs-3.6.9-1.6.1-snap-dev:1.0.1
#runtime_timeout: 600
#runtime_memory: 256
#workers: <MAX_NUM_OF_WORKERS>
#data_limit: 4  # in MiB

ibm:

iam_api_key:

code_engine: kubectl_config: /xxxx/lithops-serving-7d4a69ae-5a89-4fc2-8335-f125ec0fda87.yaml

serverless: backend: knative

ibm_cf: endpoint : namespace : api_key :

namespace_id : # Only for IAM auth

ibm_cos: endpoint : https://s3.us-south.cloud-object-storage.appdomain.cloud private_endpoint: https://s3.private.us-south.cloud-object-storage.appdomain.cloud api_key : [my api key]

knative:

istio_endpoint :

docker_user : kserving

docker_token : 


2. Run `lithops test` and got error as below

lithops test

Lithops v2.1.0 init for Knative ExecutorID 0e71a5/0 | JobID A000 - Selected Runtime: kserving/lithops-knative-v36:210 - 256MB (Installing runtime https://proxy.us-south.codeengine.cloud.ibm.com/7d4a69ae-5a89/kserving--lithops-knative-v36--210--256mb...) Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/urllib3/response.py", line 696, in _update_chunk_length self.chunk_left = int(line, 16) ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/urllib3/response.py", line 436, in _error_catcher yield File "/usr/local/lib/python3.6/dist-packages/urllib3/response.py", line 763, in read_chunked self._update_chunk_length() File "/usr/local/lib/python3.6/dist-packages/urllib3/response.py", line 700, in _update_chunk_length raise httplib.IncompleteRead(line) http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/bin/lithops", line 11, in sys.exit(cli()) File "/usr/lib/python3/dist-packages/click/core.py", line 722, in call return self.main(args, kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 535, in invoke return callback(args, **kwargs) File "/usr/local/lib/python3.6/dist-packages/lithops/cli/cli.py", line 34, in test_function pw.call_async(hello, 'World') File "/usr/local/lib/python3.6/dist-packages/lithops/executor.py", line 135, in call_async runtime_meta = self.invoker.select_runtime(job_id, runtime_memory) File "/usr/local/lib/python3.6/dist-packages/lithops/invoker.py", line 129, in select_runtime runtime_meta = compute_handler.create_runtime(runtime_name, runtime_memory, timeout=timeout) File "/usr/local/lib/python3.6/dist-packages/lithops/compute/compute.py", line 76, in create_runtime return self.compute_handler.create_runtime(runtime_name, memory, timeout=timeout) File "/usr/local/lib/python3.6/dist-packages/lithops/compute/backends/knative/knative.py", line 465, in create_runtime self._create_service(docker_image_name, memory, timeout) File "/usr/local/lib/python3.6/dist-packages/lithops/compute/backends/knative/knative.py", line 411, in _create_service timeout_seconds=300): File "/usr/local/lib/python3.6/dist-packages/kubernetes/watch/watch.py", line 144, in stream for line in iter_resp_lines(resp): File "/usr/local/lib/python3.6/dist-packages/kubernetes/watch/watch.py", line 46, in iter_resp_lines for seg in resp.read_chunked(decode_content=False): File "/usr/local/lib/python3.6/dist-packages/urllib3/response.py", line 792, in read_chunked self._original_response.close() File "/usr/lib/python3.6/contextlib.py", line 99, in exit self.gen.throw(type, value, traceback) File "/usr/local/lib/python3.6/dist-packages/urllib3/response.py", line 454, in _error_catcher raise ProtocolError("Connection broken: %r" % e, e) urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

gilv commented 3 years ago

@zhanggbj I think this is wrong configuration, old one. We now have different format. @JosepSampe right?

gilv commented 3 years ago

@zhanggbj follow https://github.com/lithops-cloud/lithops/blob/master/config/config_template.yaml

JosepSampe commented 3 years ago

I will check it, version 2.1.0 uses old config, so the problem comes from another part

aitorarjona commented 3 years ago

Last time I tried to run Lithops on Code Engine with Knative I had some trouble with the endpoint. The endpoint that Lithops retrieves throught kubernetes api starts with http, but Code Engine currently only works using https endpoint and drops trafic that use http instead of redirecting it. Could this be the problem @JosepSampe ? Although reading the stack trace I see that it fails when creating the runtime, while the issue I comment occurs when trying to invoke.

JosepSampe commented 3 years ago

Seems now knative apiVersion v1alpha1 we use in lithops is deprecated. With v1 all is working properly, I will push the fix

zhanggbj commented 3 years ago

Thanks @gilv @JosepSampe @aitorarjona for your quick response! I tried again with lithops 2.2 and also got some debug info. Could you please help take a look? Thanks!

  1. Upgrade to lithops 2.2

  2. Applied config as below

    
lithops:
#storage: ibm_cos
storage_bucket: lithops-serving
#executor: serverless
#rabbitmq_monitor: <True/False>
#data_cleaner: <True/False>
#workers: <MAX_NUM_OF_WORKERS>
#data_limit: 4  # in MiB
#include_modules: <LIST_OF_MODULES>
#exclude_modules: <LIST_OF_MODULES>

serverless: backend: knative

code_engine: kubectl_config: /root/.bluemix/plugins/code-engine/lithops-serving-7d4a69ae-5a89-4fc2-8335-f125ec0fda87.yaml

ibm_cos: endpoint : https://s3.us-south.cloud-object-storage.appdomain.cloud private_endpoint: https://s3.private.us-south.cloud-object-storage.appdomain.cloud api_key : {my api key}

knative:

istio_endpoint :

docker_user : kserving

docker_token : 


3. **lithops test --debug** got error as below

lithops test --debug INFO:lithops.config:Lithops v2.2.0 DEBUG:lithops.config:Loading configuration INFO:lithops.config:Getting configuration from /root/workspace/flops/config DEBUG:lithops.config:Loading Serverless backend module: knative DEBUG:lithops.config:Loading Storage backend module: ibm_cos INFO:lithops.config:Lithops v2.2.0 DEBUG:lithops.config:Loading configuration DEBUG:lithops.config:Loading Serverless backend module: knative DEBUG:lithops.config:Loading Storage backend module: ibm_cos DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:Creating IBM COS client DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:Set IBM COS Endpoint to https://s3.us-south.cloud-object-storage.appdomain.cloud DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:Using IBM COS API Key - Reusing Token from local cache DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:Token expiry time: 2020-10-22 02:33:18.782869-04:00 - Minutes left: 6 DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:Using IBM COS API Key - Token expired. Requesting new token DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:IBM COS client created successfully INFO:lithops.serverless.backends.knative.knative:istio-ingressgateway endpoint not found DEBUG:lithops.serverless.backends.knative.knative:Loaded service host suffix: None INFO:lithops.serverless.backends.knative.knative:Lithops v2.2.0 init for Knative DEBUG:lithops.invokers:ExecutorID b156f1/0 - Total available workers: 100 DEBUG:lithops.invokers:ExecutorID b156f1/0 - Serverless invoker created INFO:lithops.executors:Serverless Executor created with ID: b156f1/0 INFO:lithops.invokers:ExecutorID b156f1/0 | JobID A000 - Selected Runtime: kserving/lithops-knative-v36:220 - 256MB DEBUG:lithops.storage.storage:Runtime metadata not found in local cache. Retrieving it from storage DEBUG:lithops.storage.storage:Trying to download runtime metadata from: ibm_cos://lithops-serving/lithops.runtimes/2.2.0/https://proxy.us-south.codeengine.cloud.ibm.com/7d4a69ae-5a89/kserving--lithops-knative-v36--220--256mb.meta.json DEBUG:lithops.storage.storage:Runtime metadata not found in storage DEBUG:lithops.invokers:ExecutorID b156f1/0 | JobID A000 - Runtime kserving/lithops-knative-v36:220 with 256MB is not yet installed DEBUG:lithops.invokers:Creating runtime: kserving/lithops-knative-v36:220, memory: 256MB INFO:lithops.serverless.backends.knative.knative:Building a new docker image from Dockerfile INFO:lithops.serverless.backends.knative.knative:Docker image name: kserving/lithops-knative-v36:220 DEBUG:lithops.serverless.utils:Creating function handler zip in /root/lithops_knative.zip Sending build context to Docker daemon 31.72GB Step 1/8 : FROM python:3.6-slim-buster ---> f9fcc0353bbd Step 2/8 : RUN apt-get update && apt-get install -y zip && rm -rf /var/lib/apt/lists/ ---> Using cache ---> 9c0b6ad71f33 Step 3/8 : RUN pip install --upgrade setuptools six pip && pip install --no-cache-dir gunicorn==19.9.0 pika==0.13.1 flask gevent glob2 ibm-cos-sdk redis requests PyYAML kubernetes numpy ---> Using cache ---> f5d56096f2b9 Step 4/8 : ENV APP_HOME /lithops ---> Using cache ---> 5f13db7cab21 Step 5/8 : WORKDIR $APP_HOME ---> Using cache ---> a049373d7085 Step 6/8 : COPY lithops_knative.zip . ---> Using cache ---> afe780d3b48a Step 7/8 : RUN unzip lithops_knative.zip && rm lithops_knative.zip ---> Using cache ---> 25a5aa73417d Step 8/8 : CMD exec gunicorn --bind :$PORT --workers 1 --timeout 600 lithopsproxy:proxy ---> Using cache ---> 004e52f393c7 Successfully built 004e52f393c7 Successfully tagged kserving/lithops-knative-v36:220 The push refers to repository [docker.io/kserving/lithops-knative-v36] 2934b7fccfdb: Layer already exists 37972020c27e: Layer already exists 5229f161a2d0: Layer already exists 8723d030b279: Layer already exists 8969d0d592f3: Layer already exists c04dd0e99e62: Layer already exists 2aa642c0a1e4: Layer already exists bbc6d16a112a: Layer already exists 225ef82ca30a: Layer already exists d0fe97fa8b8c: Layer already exists 220: digest: sha256:32b28f3d8c2aa6dc7752f5fcb56d57d4b5dea9f191758febb0ef81e8a92097df size: 2419 DEBUG:lithops.serverless.backends.knative.knative:Creating Lithops runtime service in Knative DEBUG:lithops.serverless.backends.knative.knative:Service name: kserving--lithops-knative-v36--220--256mb DEBUG:lithops.serverless.backends.knative.knative:Namespace: 7d4a69ae-5a89 Traceback (most recent call last): File "/usr/local/bin/lithops", line 11, in sys.exit(cli()) File "/usr/lib/python3/dist-packages/click/core.py", line 722, in call return self.main(args, kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 535, in invoke return callback(*args, kwargs) File "/usr/local/lib/python3.6/dist-packages/lithops/cli/cli.py", line 34, in test_function pw.call_async(hello, 'World') File "/usr/local/lib/python3.6/dist-packages/lithops/executors.py", line 156, in call_async runtime_meta = self.invoker.select_runtime(job_id, runtime_memory) File "/usr/local/lib/python3.6/dist-packages/lithops/invokers.py", line 111, in select_runtime runtime_meta = self.compute_handler.create_runtime(runtime_name, runtime_memory, timeout=timeout) File "/usr/local/lib/python3.6/dist-packages/lithops/serverless/serverless.py", line 62, in create_runtime return self.backend.create_runtime(runtime_name, memory, timeout=timeout) File "/usr/local/lib/python3.6/dist-packages/lithops/serverless/backends/knative/knative.py", line 465, in create_runtime self._create_service(docker_image_name, memory, timeout) File "/usr/local/lib/python3.6/dist-packages/lithops/serverless/backends/knative/knative.py", line 403, in _create_service body=svc_res File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api/custom_objects_api.py", line 183, in create_namespaced_custom_object (data) = self.create_namespaced_custom_object_with_http_info(group, version, namespace, plural, body, kwargs) # noqa: E501 File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api/custom_objects_api.py", line 289, in create_namespaced_custom_object_with_http_info collection_formats=collection_formats) File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 345, in call_api _preload_content, _request_timeout) File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 176, in __call_api _request_timeout=_request_timeout) File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/api_client.py", line 388, in request body=body) File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/rest.py", line 278, in POST body=body) File "/usr/local/lib/python3.6/dist-packages/kubernetes/client/rest.py", line 231, in request raise ApiException(http_resp=r) kubernetes.client.rest.ApiException: (401) Reason: Unauthorized HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 22 Oct 2020 06:53:08 GMT', 'Content-Type': 'application/json', 'Content-Length': '129', 'Connection': 'keep-alive', 'X-Content-Type-Options': 'nosniff', 'Content-Security-Policy': "default-src 'self'", 'X-XSS-Protection': '1; mode=block', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'}) HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

3. Run **lithops verify**, it repeated the following info.

lithops verify -d | tee lithops.verify INFO:lithops.config:Lithops v2.2.0 DEBUG:lithops.config:Loading configuration INFO:lithops.config:Getting configuration from /root/workspace/flops/config DEBUG:lithops.config:Loading Serverless backend module: knative DEBUG:lithops.config:Loading Storage backend module: ibm_cos DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:Creating IBM COS client DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:Set IBM COS Endpoint to https://s3.us-south.cloud-object-storage.appdomain.cloud DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:Using IBM COS API Key - Reusing Token from local cache DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:Token expiry time: 2020-10-22 02:33:18.782869-04:00 - Minutes left: 18 DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:IBM COS client created successfully DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:PUT Object lithops.test/test2 - Size: 98.4KiB - OK DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:PUT Object lithops.test/test0 - Size: 230.6KiB - OK DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:PUT Object lithops.test/test1 - Size: 54.2KiB - OK DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:PUT Object lithops.test/test3 - Size: 1.3MiB - OK DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:PUT Object __lithops.test/test4 - Size: 1.4MiB - OK INFO:lithops.config:Lithops v2.2.0 DEBUG:lithops.config:Loading configuration DEBUG:lithops.config:Loading Serverless backend module: knative DEBUG:lithops.config:Loading Storage backend module: ibm_cos INFO:lithops.config:Lithops v2.2.0 DEBUG:lithops.config:Loading configuration DEBUG:lithops.config:Loading Serverless backend module: knative DEBUG:lithops.config:Loading Storage backend module: ibm_cos DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:Creating IBM COS client DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:Set IBM COS Endpoint to https://s3.us-south.cloud-object-storage.appdomain.cloud DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:Using IBM COS API Key - Reusing Token from config DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:Token expiry time: 2020-10-22 02:33:18.782869-04:00 - Minutes left: 18 DEBUG:lithops.storage.backends.ibm_cos.ibm_cos:IBM COS client created successfully INFO:lithops.serverless.backends.knative.knative:istio-ingressgateway endpoint not found DEBUG:lithops.serverless.backends.knative.knative:Loaded service host suffix: None INFO:lithops.serverless.backends.knative.knative:Lithops v2.2.0 init for Knative DEBUG:lithops.invokers:ExecutorID 58101f/0 - Total available workers: 100 DEBUG:lithops.invokers:ExecutorID 58101f/0 - Serverless invoker created INFO:lithops.executors:Serverless Executor created with ID: 58101f/0 INFO:lithops.invokers:ExecutorID 58101f/0 | JobID A000 - Selected Runtime: kserving/lithops-knative-v36:220 - 256MB DEBUG:lithops.storage.storage:Runtime metadata not found in local cache. Retrieving it from storage DEBUG:lithops.storage.storage:Trying to download runtime metadata from: ibm_cos://lithops-serving/lithops.runtimes/2.2.0/https://proxy.us-south.codeengine.cloud.ibm.com/7d4a69ae-5a89/kserving--lithops-knative-v36--220--256mb.meta.json DEBUG:lithops.storage.storage:Runtime metadata not found in storage DEBUG:lithops.invokers:ExecutorID 58101f/0 | JobID A000 - Runtime kserving/lithops-knative-v36:220 with 256MB is not yet installed DEBUG:lithops.invokers:Creating runtime: kserving/lithops-knative-v36:220, memory: 256MB INFO:lithops.serverless.backends.knative.knative:Building a new docker image from Dockerfile INFO:lithops.serverless.backends.knative.knative:Docker image name: kserving/lithops-knative-v36:220 DEBUG:lithops.serverless.utils:Creating function handler zip in /root/workspace/flops/lithops_knative.zip Sending build context to Docker daemon 320kB Step 1/8 : FROM python:3.6-slim-buster ---> f9fcc0353bbd Step 2/8 : RUN apt-get update && apt-get install -y zip && rm -rf /var/lib/apt/lists/* ---> Using cache ---> 9c0b6ad71f33 Step 3/8 : RUN pip install --upgrade setuptools six pip && pip install --no-cache-dir gunicorn==19.9.0 pika==0.13.1 flask gevent glob2 ibm-cos-sdk redis requests PyYAML kubernetes numpy ---> Using cache ---> f5d56096f2b9 Step 4/8 : ENV APP_HOME /lithops ---> Using cache ---> 5f13db7cab21 Step 5/8 : WORKDIR $APP_HOME ---> Using cache ---> a049373d7085 Step 6/8 : COPY lithops_knative.zip . ---> Using cache ---> afe780d3b48a Step 7/8 : RUN unzip lithops_knative.zip && rm lithops_knative.zip ---> Using cache ---> 25a5aa73417d Step 8/8 : CMD exec gunicorn --bind :$PORT --workers 1 --timeout 600 lithopsproxy:proxy ---> Using cache ---> 004e52f393c7 Successfully built 004e52f393c7 Successfully tagged kserving/lithops-knative-v36:220 The push refers to repository [docker.io/kserving/lithops-knative-v36] 2934b7fccfdb: Preparing 37972020c27e: Preparing 5229f161a2d0: Preparing 8723d030b279: Preparing 8969d0d592f3: Preparing c04dd0e99e62: Preparing 2aa642c0a1e4: Preparing bbc6d16a112a: Preparing 225ef82ca30a: Preparing d0fe97fa8b8c: Preparing 2aa642c0a1e4: Waiting bbc6d16a112a: Waiting 225ef82ca30a: Waiting d0fe97fa8b8c: Waiting c04dd0e99e62: Waiting 8723d030b279: Layer already exists 37972020c27e: Layer already exists 8969d0d592f3: Layer already exists 2934b7fccfdb: Layer already exists 5229f161a2d0: Layer already exists bbc6d16a112a: Layer already exists c04dd0e99e62: Layer already exists 225ef82ca30a: Layer already exists d0fe97fa8b8c: Layer already exists 2aa642c0a1e4: Layer already exists 220: digest: sha256:32b28f3d8c2aa6dc7752f5fcb56d57d4b5dea9f191758febb0ef81e8a92097df size: 2419 DEBUG:lithops.serverless.backends.knative.knative:Creating Lithops runtime service in Knative DEBUG:lithops.serverless.backends.knative.knative:Service name: kserving--lithops-knative-v36--220--256mb DEBUG:lithops.serverless.backends.knative.knative:Namespace: 7d4a69ae-5a89

JosepSampe commented 3 years ago

Version 2.2.0 has the same issue. I added the fix here: #393 @gilv Could you check it and merge? @zhanggbj Once merged, could you try with master branch?

zhanggbj commented 3 years ago

@JosepSampe Sure, after #393 is merged I can give a try.

JosepSampe commented 3 years ago

@zhanggbj merged, you can already test it

zhanggbj commented 3 years ago

@JosepSampe @gilv After build and install master branch, lithops test is working now! Thanks!

[root@zhanggbjwsp ~ (āŽˆ |7d4a69ae-5a89:7d4a69ae-5a89)]# lithops test
Lithops v2.2.1dev init for Knative
ExecutorID e2f599/0 | JobID A000 - Selected Runtime: kserving/lithops-knative-v36:221dev - 256MB
ExecutorID e2f599/0 | JobID A000 - Uploading function and data - Total: 629.0B
ExecutorID e2f599/0 | JobID A000 - Starting function invocation: hello()  - Total: 1 activations
ExecutorID e2f599/0 - Getting results...

  100%|ā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆā–ˆ| 1/1

ExecutorID e2f599/0 - Cleaning temporary data

Hello World! Lithops is working as expected :)