lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
http://lithops.cloud
Apache License 2.0
320 stars 106 forks source link

can't parse kubeconfig correctly with lithops 2.2.13 #489

Closed shiyan0109 closed 3 years ago

shiyan0109 commented 3 years ago

I have a simple code which run on lithops 2.2.9 + runtime ibmfunctions/lithops-ce-3.8.5-2.2.9:1.0.0 successfully.

code is

import lithops

iterdata = ['Gil', 'Dana', 'John', 'Scott']

def add_value(name):
    return 'Hello ' + name

if __name__ == '__main__':
    lt = lithops.FunctionExecutor(mode="serverless",
            backend='code_engine',
            log_level='DEBUG')
    lt.map(add_value,  iterdata)
    print (lt.get_result())

lithops_config file is

lithops: 
    storage: ibm_cos
    storage_bucket: lithops-batch

    mode: serverless
    workers: 1000

serverless:
    backend: code_engine
    runtime_memory: 256
    runtime : ibmfunctions/lithops-codeengine-v38:2213

code_engine:
    kubectl_config: ~/.kube/config
    docker_user: my-name
    runtime_cpu: 200m

ibm_cos:
    endpoint    : https://s3.us-south.cloud-object-storage.appdomain.cloud
    private_endpoint: https://s3.private.us-south.cloud-object-storage.appdomain.cloud
    api_key     : xx

Executed successfully.

2020-12-10 15:51:08,306 [INFO] lithops.config -- Lithops v2.2.13
2020-12-10 15:51:08,306 [DEBUG] lithops.config -- Loading configuration
2020-12-10 15:51:08,306 [DEBUG] lithops.config -- Getting configuration from /Users/shiyan/work/coligo/lithops/.lithops_config
2020-12-10 15:51:08,327 [DEBUG] lithops.config -- Loading Serverless backend module: code_engine
2020-12-10 15:51:08,707 [DEBUG] lithops.config -- Loading Storage backend module: ibm_cos
2020-12-10 15:51:08,764 [DEBUG] lithops.storage.backends.ibm_cos.ibm_cos -- Creating IBM COS client
2020-12-10 15:51:08,764 [DEBUG] lithops.storage.backends.ibm_cos.ibm_cos -- Set IBM COS Endpoint to https://s3.us-south.cloud-object-storage.appdomain.cloud
2020-12-10 15:51:08,765 [DEBUG] lithops.util.ibm_token_manager -- Using IBM COS API Key - Reusing Token from local cache
2020-12-10 15:51:08,768 [DEBUG] lithops.util.ibm_token_manager -- Token expiry time: 2020-12-10 16:50:56.863872+08:00 - Minutes left: 59
2020-12-10 15:51:08,816 [INFO] lithops.storage.backends.ibm_cos.ibm_cos -- Created IBM COS Storage client - Region: us-south
2020-12-10 15:51:08,816 [DEBUG] lithops.serverless.backends.code_engine.code_engine -- Creating IBM Code Engine client
2020-12-10 15:51:08,852 [ERROR] lithops.serverless.serverless -- There was an error trying to create the code_engine serverless backend
Traceback (most recent call last):
  File "first-test.py", line 9, in <module>
    lt = lithops.FunctionExecutor(mode="serverless",
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/lithops/executors.py", line 127, in __init__
    self.compute_handler = ServerlessHandler(serverless_config,
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/lithops/serverless/serverless.py", line 44, in __init__
    raise e
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/lithops/serverless/serverless.py", line 39, in __init__
    self.backend = ServerlessBackend(self.config[self.backend_name], storage_config)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/lithops/serverless/backends/code_engine/code_engine.py", line 65, in __init__
    self.region = self.cluster.split('//')[1].split('.')[1]
IndexError: list index out of range

Anything change on this part? my kubernetst version is 1.18 (both server and client)

gilv commented 3 years ago

@JosepSampe @kpavel it fails on reading some value from kube config file. Do you know what is wrong? For me it working, for some not...perhaps this value not always present?

JosepSampe commented 3 years ago

Fix added in #490

JosepSampe commented 3 years ago

@shiyan0109 just out of curiosity, what is the value of the cluster var you obtain in your kubeconfig file?

sven-lange-last commented 3 years ago

@JosepSampe please do not expect any particular format for the value of cluster. This highly depends on the component creating the Kubeconfig.

When we are working on Code Engine clusters as developers, the format is <cluster name>/<cluster id>. When we are using KinD clusters, it's <cluster name>.