lithops-cloud / lithops

A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
http://lithops.cloud
Apache License 2.0
317 stars 105 forks source link

Runtime_timeout is not set correctly #976

Closed ShuxinLin closed 2 years ago

ShuxinLin commented 2 years ago

Hi I am running lithops with code engine. I set runtime_timeout and execution_timeout in the cofig to be both 21600 seconds (6 hours). But the execution_timeout will be always set to 1195 because of this line https://github.com/lithops-cloud/lithops/blob/09d3181043cc782916675e59cda463b3895ab38c/lithops/job/job.py#L208

So I decide to print the runtime_meta, and here is the content

{'preinstalls': [['OpenSSL', True], ['PIL', True], ['__future__', False], ['_asyncio', False], ['_bisect', False], ['_blake2', False], ['_bootlocale', False], ['_bz2', False], ['_cffi_backend', False], ['_codecs_cn', False], ['_codecs_hk', False], ['_codecs_iso2022', False], ['_codecs_jp', False], ['_codecs_kr', False], ['_codecs_tw', False], ['_collections_abc', False], ['_compat_pickle', False], ['_compression', False], ['_contextvars', False], ['_crypt', False], ['_csv', False], ['_ctypes', False], ['_ctypes_test', False], ['_curses', False], ['_curses_panel', False], ['_datetime', False], ['_decimal', False], ['_distutils_hack', True], ['_dummy_thread', False], ['_elementtree', False], ['_gdbm', False], ['_hashlib', False], ['_heapq', False], ['_json', False], ['_lsprof', False], ['_lzma', False], ['_markupbase', False], ['_md5', False], ['_multibytecodec', False], ['_multiprocessing', False], ['_opcode', False], ['_osx_support', False], ['_pickle', False], ['_posixsubprocess', False], ['_py_abc', False], ['_pydecimal', False], ['_pyio', False], ['_queue', False], ['_random', False], ['_sha1', False], ['_sha256', False], ['_sha3', False], ['_sha512', False], ['_sitebuiltins', False], ['_socket', False], ['_sqlite3', False], ['_ssl', False], ['_strptime', False], ['_struct', False], ['_sysconfigdata_m_linux_x86_64-linux-gnu', False], ['_testbuffer', False], ['_testcapi', False], ['_testimportmultiple', False], ['_testmultiphase', False], ['_threading_local', False], ['_tkinter', False], ['_uuid', False], ['_weakrefset', False], ['_xxtestfuzz', False], ['_yaml', True], ['abc', False], ['absl', True], ['aifc', False], ['antigravity', False], ['argparse', False], ['array', False], ['ast', False], ['astunparse', True], ['async_timeout', True], ['asynchat', False], ['asyncio', True], ['asyncore', False], ['attr', True], ['attrs', True], ['audioop', False], ['automat', True], ['base64', False], ['bcrypt', True], ['bdb', False], ['binascii', False], ['binhex', False], ['bisect', False], ['bs4', True], ['bz2', False], ['cProfile', False], ['cachetools', True], ['calendar', False], ['certifi', True], ['cffi', True], ['cgi', False], ['cgitb', False], ['charset_normalizer', True], ['chunk', False], ['clang', True], ['click', True], ['cloudpickle', True], ['cmath', False], ['cmd', False], ['code', False], ['codecs', False], ['codeop', False], ['collections', True], ['colorsys', False], ['compileall', False], ['concurrent', True], ['configparser', False], ['constantly', True], ['contextlib', False], ['contextvars', False], ['copy', False], ['copyreg', False], ['crypt', False], ['cryptography', True], ['cssselect', True], ['csv', False], ['ctypes', True], ['curses', True], ['cycler', False], ['dataclasses', False], ['datetime', False], ['dateutil', True], ['dbm', True], ['decimal', False], ['deprecated', True], ['difflib', False], ['dill', True], ['dis', False], ['distutils', True], ['docker', True], ['doctest', False], ['docutils', True], ['dummy_threading', False], ['email', True], ['encodings', True], ['ensurepip', True], ['enum', False], ['etcd3', True], ['fcntl', False], ['filecmp', False], ['fileinput', False], ['filelock', True], ['flask', True], ['flatbuffers', True], ['fnmatch', False], ['fontTools', True], ['formatter', False], ['fractions', False], ['ftplib', False], ['functools', False], ['gast', True], ['genericpath', False], ['getopt', False], ['getpass', False], ['gettext', False], ['gevent', True], ['glob', False], ['google_auth_oauthlib', True], ['greenlet', True], ['grp', False], ['grpc', True], ['gunicorn', True], ['gzip', False], ['h5py', True], ['hashlib', False], ['heapq', False], ['hmac', False], ['html', True], ['http', True], ['httplib2', True], ['hyperlink', True], ['ibm_boto3', True], ['ibm_botocore', True], ['ibm_cloud_sdk_core', True], ['ibm_s3transfer', True], ['ibm_vpc', True], ['ibm_watson_machine_learning', True], ['ibmfl', True], ['idlelib', True], ['idna', True], ['imaplib', False], ['imblearn', True], ['imghdr', False], ['imp', False], ['importlib', True], ['importlib_metadata', True], ['incremental', True], ['inspect', False], ['io', False], ['ipaddress', False], ['isympy', False], ['itemadapter', True], ['itemloaders', True], ['itsdangerous', True], ['jinja2', True], ['jmespath', True], ['joblib', True], ['json', True], ['jwt', True], ['kafka', True], ['keras', True], ['keras_preprocessing', True], ['keyword', False], ['kiwisolver', True], ['kubernetes', True], ['lib2to3', True], ['lifelines', True], ['lightgbm', True], ['linecache', False], ['lithops', True], ['lithopsentry', False], ['locale', False], ['logging', True], ['lomond', True], ['lxml', True], ['lzma', False], ['macpath', False], ['mailbox', False], ['mailcap', False], ['markdown', True], ['markupsafe', True], ['math', False], ['matplotlib', True], ['mimetypes', False], ['mmap', False], ['modulefinder', False], ['mpmath', True], ['multiprocessing', True], ['nacl', True], ['netrc', False], ['networkx', True], ['nis', False], ['nntplib', False], ['ntpath', False], ['nturl2path', False], ['numbers', False], ['numpy', True], ['oauthlib', True], ['opcode', False], ['operator', False], ['opt_einsum', True], ['optparse', False], ['os', False], ['ossaudiodev', False], ['outliers', True], ['packaging', True], ['pandas', True], ['paramiko', True], ['parsel', True], ['parser', False], ['pasta', True], ['pathlib', False], ['patsy', True], ['pdb', False], ['pickle', False], ['pickletools', False], ['pika', True], ['pip', True], ['pipes', False], ['pkg_resources', True], ['pkgutil', False], ['platform', False], ['plistlib', False], ['poplib', False], ['posixpath', False], ['pprint', False], ['profile', False], ['protego', False], ['ps_mem', False], ['pstats', False], ['psycopg2', True], ['pty', False], ['py_compile', False], ['pyasn1', True], ['pyasn1_modules', True], ['pyclbr', False], ['pycparser', True], ['pydispatch', True], ['pydoc', False], ['pydoc_data', True], ['pyexpat', False], ['pylab', False], ['pyparsing', True], ['pytz', True], ['pywt', True], ['queue', False], ['queuelib', True], ['quopri', False], ['random', False], ['re', False], ['readline', False], ['redis', True], ['reprlib', False], ['requests', True], ['requests_file', False], ['requests_oauthlib', True], ['resource', False], ['retrying', False], ['rlcompleter', False], ['rsa', True], ['runpy', False], ['sched', False], ['scipy', True], ['scrapy', True], ['seaborn', True], ['secrets', False], ['select', False], ['selectors', False], ['service_identity', True], ['setuptools', True], ['shelve', False], ['shlex', False], ['shutil', False], ['signal', False], ['simplejson', True], ['site', False], ['six', False], ['sklearn', True], ['smtpd', False], ['smtplib', False], ['sndhdr', False], ['socket', False], ['socketserver', False], ['soupsieve', True], ['spwd', False], ['sqlite3', True], ['sre_compile', False], ['sre_constants', False], ['sre_parse', False], ['srom', True], ['ssl', False], ['stat', False], ['statistics', False], ['statsmodels', True], ['string', False], ['stringprep', False], ['struct', False], ['subprocess', False], ['sunau', False], ['symbol', False], ['sympy', True], ['symtable', False], ['sysconfig', False], ['syslog', False], ['tabnanny', False], ['tabulate', False], ['tarfile', False], ['tblib', True], ['telnetlib', False], ['tempfile', False], ['tenacity', True], ['tensorboard', True], ['tensorboard_data_server', True], ['tensorboard_plugin_wit', True], ['tensorflow', True], ['tensorflow_estimator', True], ['tensorflow_io_gcs_filesystem', True], ['termcolor', False], ['termios', False], ['test', True], ['test_integration', True], ['textwrap', False], ['this', False], ['threading', False], ['threadpoolctl', False], ['timeit', False], ['tkinter', True], ['tldextract', True], ['token', False], ['tokenize', False], ['tqdm', True], ['trace', False], ['traceback', False], ['tracemalloc', False], ['tty', False], ['turtle', False], ['turtledemo', True], ['twisted', True], ['types', False], ['typing', False], ['typing_extensions', False], ['unicodedata', False], ['unittest', True], ['urllib', True], ['urllib3', True], ['uu', False], ['uuid', False], ['venv', True], ['w3lib', True], ['warnings', False], ['wave', False], ['weakref', False], ['webbrowser', False], ['websocket', True], ['werkzeug', True], ['wheel', True], ['wrapt', True], ['wsgiref', True], ['xdrlib', False], ['xgboost', True], ['xml', True], ['xmlrpc', True], ['xxlimited', False], ['yaml', True], ['zipapp', False], ['zipfile', False], ['zipp', False], ['zlib', False]], 'python_version': '3.7', 'lithops_version': '2.6.1.dev0', 'runtime_timeout': 1200}

Where does runtine_timeout=1200 come from? I guess it is some default values?

JosepSampe commented 2 years ago

In principle we are not forcing 1200 seconds anywhere. May you have it in your config file? I just ran a test and the timeout is set correctly using master branch

image image

ShuxinLin commented 2 years ago

@JosepSampe I found this weird bug. Here is my script

import lithops

def my_function(x):
    import time
    time.sleep(1300)

    return x + 7

config = {}         # set execution_timeout and runtime_timeout to be both 21600

fexec = lithops.FunctionExecutor(
    mode="serverless", backend="code_engine", config=config
)

fexec.call_async(my_function, 3)
response = fexec.get_result()
print(response)

In the lithops source code here, https://github.com/lithops-cloud/lithops/blob/09d3181043cc782916675e59cda463b3895ab38c/lithops/job/job.py#L199 I print some variables like this

    if mode == SERVERLESS:
        job.runtime_memory = runtime_memory or config[backend]['runtime_memory']
        job.runtime_timeout = runtime_meta['runtime_timeout']
        print("runtime_meta", runtime_meta)
        print("job.execution_timeout", job.execution_timeout)

        if job.execution_timeout >= job.runtime_timeout:
            job.execution_timeout = job.runtime_timeout - 5
        print('after:')
        print("job.execution_timeout", job.execution_timeout)
        print("job.runtime_timeout", job.runtime_timeout)

I installed lithops latest master

This is the output

/Users/shuxin.lin/miniconda/envs/py37/bin/python /Users/shuxin.lin/Projects/srom-anomaly-service/celery-queue/main.py
2022-07-28 15:27:51,365 [INFO] config.py:131 -- Lithops v2.6.1.dev0
2022-07-28 15:27:52,255 [INFO] ibm_cos.py:98 -- IBM COS client created - Region: us-south
2022-07-28 15:27:52,259 [INFO] code_engine.py:131 -- IBM Code Engine client created - Region: us-south
2022-07-28 15:27:52,259 [INFO] invokers.py:108 -- ExecutorID 1006d1-0 | JobID A000 - Selected Runtime: us.icr.io/srom-anomaly/lithops-srom:1.5.9.dev0 - 16384MB
runtime_meta {'preinstalls': [['OpenSSL', True], ['PIL', True], ['__future__', False], ['_asyncio', False], ['_bisect', False], ['_blake2', False], ['_bootlocale', False], ['_bz2', False], ['_cffi_backend', False], ['_codecs_cn', False], ['_codecs_hk', False], ['_codecs_iso2022', False], ['_codecs_jp', False], ['_codecs_kr', False], ['_codecs_tw', False], ['_collections_abc', False], ['_compat_pickle', False], ['_compression', False], ['_contextvars', False], ['_crypt', False], ['_csv', False], ['_ctypes', False], ['_ctypes_test', False], ['_curses', False], ['_curses_panel', False], ['_datetime', False], ['_decimal', False], ['_distutils_hack', True], ['_dummy_thread', False], ['_elementtree', False], ['_gdbm', False], ['_hashlib', False], ['_heapq', False], ['_json', False], ['_lsprof', False], ['_lzma', False], ['_markupbase', False], ['_md5', False], ['_multibytecodec', False], ['_multiprocessing', False], ['_opcode', False], ['_osx_support', False], ['_pickle', False], ['_posixsubprocess', False], ['_py_abc', False], ['_pydecimal', False], ['_pyio', False], ['_queue', False], ['_random', False], ['_sha1', False], ['_sha256', False], ['_sha3', False], ['_sha512', False], ['_sitebuiltins', False], ['_socket', False], ['_sqlite3', False], ['_ssl', False], ['_strptime', False], ['_struct', False], ['_sysconfigdata_m_linux_x86_64-linux-gnu', False], ['_testbuffer', False], ['_testcapi', False], ['_testimportmultiple', False], ['_testmultiphase', False], ['_threading_local', False], ['_tkinter', False], ['_uuid', False], ['_weakrefset', False], ['_xxtestfuzz', False], ['_yaml', True], ['abc', False], ['absl', True], ['aifc', False], ['antigravity', False], ['argparse', False], ['array', False], ['ast', False], ['astunparse', True], ['async_timeout', True], ['asynchat', False], ['asyncio', True], ['asyncore', False], ['attr', True], ['attrs', True], ['audioop', False], ['automat', True], ['base64', False], ['bcrypt', True], ['bdb', False], ['binascii', False], ['binhex', False], ['bisect', False], ['bs4', True], ['bz2', False], ['cProfile', False], ['cachetools', True], ['calendar', False], ['certifi', True], ['cffi', True], ['cgi', False], ['cgitb', False], ['charset_normalizer', True], ['chunk', False], ['clang', True], ['click', True], ['cloudpickle', True], ['cmath', False], ['cmd', False], ['code', False], ['codecs', False], ['codeop', False], ['collections', True], ['colorsys', False], ['compileall', False], ['concurrent', True], ['configparser', False], ['constantly', True], ['contextlib', False], ['contextvars', False], ['copy', False], ['copyreg', False], ['crypt', False], ['cryptography', True], ['cssselect', True], ['csv', False], ['ctypes', True], ['curses', True], ['cycler', False], ['dataclasses', False], ['datetime', False], ['dateutil', True], ['dbm', True], ['decimal', False], ['deprecated', True], ['difflib', False], ['dill', True], ['dis', False], ['distutils', True], ['docker', True], ['doctest', False], ['docutils', True], ['dummy_threading', False], ['email', True], ['encodings', True], ['ensurepip', True], ['enum', False], ['etcd3', True], ['fcntl', False], ['filecmp', False], ['fileinput', False], ['filelock', True], ['flask', True], ['flatbuffers', True], ['fnmatch', False], ['fontTools', True], ['formatter', False], ['fractions', False], ['ftplib', False], ['functools', False], ['gast', True], ['genericpath', False], ['getopt', False], ['getpass', False], ['gettext', False], ['gevent', True], ['glob', False], ['google_auth_oauthlib', True], ['greenlet', True], ['grp', False], ['grpc', True], ['gunicorn', True], ['gzip', False], ['h5py', True], ['hashlib', False], ['heapq', False], ['hmac', False], ['html', True], ['http', True], ['httplib2', True], ['hyperlink', True], ['ibm_boto3', True], ['ibm_botocore', True], ['ibm_cloud_sdk_core', True], ['ibm_s3transfer', True], ['ibm_vpc', True], ['ibm_watson_machine_learning', True], ['ibmfl', True], ['idlelib', True], ['idna', True], ['imaplib', False], ['imblearn', True], ['imghdr', False], ['imp', False], ['importlib', True], ['importlib_metadata', True], ['incremental', True], ['inspect', False], ['io', False], ['ipaddress', False], ['isympy', False], ['itemadapter', True], ['itemloaders', True], ['itsdangerous', True], ['jinja2', True], ['jmespath', True], ['joblib', True], ['json', True], ['jwt', True], ['kafka', True], ['keras', True], ['keras_preprocessing', True], ['keyword', False], ['kiwisolver', True], ['kubernetes', True], ['lib2to3', True], ['lifelines', True], ['lightgbm', True], ['linecache', False], ['lithops', True], ['lithopsentry', False], ['locale', False], ['logging', True], ['lomond', True], ['lxml', True], ['lzma', False], ['macpath', False], ['mailbox', False], ['mailcap', False], ['markdown', True], ['markupsafe', True], ['math', False], ['matplotlib', True], ['mimetypes', False], ['mmap', False], ['modulefinder', False], ['mpmath', True], ['multiprocessing', True], ['nacl', True], ['netrc', False], ['networkx', True], ['nis', False], ['nntplib', False], ['ntpath', False], ['nturl2path', False], ['numbers', False], ['numpy', True], ['oauthlib', True], ['opcode', False], ['operator', False], ['opt_einsum', True], ['optparse', False], ['os', False], ['ossaudiodev', False], ['outliers', True], ['packaging', True], ['pandas', True], ['paramiko', True], ['parsel', True], ['parser', False], ['pasta', True], ['pathlib', False], ['patsy', True], ['pdb', False], ['pickle', False], ['pickletools', False], ['pika', True], ['pip', True], ['pipes', False], ['pkg_resources', True], ['pkgutil', False], ['platform', False], ['plistlib', False], ['poplib', False], ['posixpath', False], ['pprint', False], ['profile', False], ['protego', False], ['ps_mem', False], ['pstats', False], ['psycopg2', True], ['pty', False], ['py_compile', False], ['pyasn1', True], ['pyasn1_modules', True], ['pyclbr', False], ['pycparser', True], ['pydispatch', True], ['pydoc', False], ['pydoc_data', True], ['pyexpat', False], ['pylab', False], ['pyparsing', True], ['pytz', True], ['pywt', True], ['queue', False], ['queuelib', True], ['quopri', False], ['random', False], ['re', False], ['readline', False], ['redis', True], ['reprlib', False], ['requests', True], ['requests_file', False], ['requests_oauthlib', True], ['resource', False], ['retrying', False], ['rlcompleter', False], ['rsa', True], ['runpy', False], ['sched', False], ['scipy', True], ['scrapy', True], ['seaborn', True], ['secrets', False], ['select', False], ['selectors', False], ['service_identity', True], ['setuptools', True], ['shelve', False], ['shlex', False], ['shutil', False], ['signal', False], ['simplejson', True], ['site', False], ['six', False], ['sklearn', True], ['smtpd', False], ['smtplib', False], ['sndhdr', False], ['socket', False], ['socketserver', False], ['soupsieve', True], ['spwd', False], ['sqlite3', True], ['sre_compile', False], ['sre_constants', False], ['sre_parse', False], ['srom', True], ['ssl', False], ['stat', False], ['statistics', False], ['statsmodels', True], ['string', False], ['stringprep', False], ['struct', False], ['subprocess', False], ['sunau', False], ['symbol', False], ['sympy', True], ['symtable', False], ['sysconfig', False], ['syslog', False], ['tabnanny', False], ['tabulate', False], ['tarfile', False], ['tblib', True], ['telnetlib', False], ['tempfile', False], ['tenacity', True], ['tensorboard', True], ['tensorboard_data_server', True], ['tensorboard_plugin_wit', True], ['tensorflow', True], ['tensorflow_estimator', True], ['tensorflow_io_gcs_filesystem', True], ['termcolor', False], ['termios', False], ['test', True], ['test_integration', True], ['textwrap', False], ['this', False], ['threading', False], ['threadpoolctl', False], ['timeit', False], ['tkinter', True], ['tldextract', True], ['token', False], ['tokenize', False], ['tqdm', True], ['trace', False], ['traceback', False], ['tracemalloc', False], ['tty', False], ['turtle', False], ['turtledemo', True], ['twisted', True], ['types', False], ['typing', False], ['typing_extensions', False], ['unicodedata', False], ['unittest', True], ['urllib', True], ['urllib3', True], ['uu', False], ['uuid', False], ['venv', True], ['w3lib', True], ['warnings', False], ['wave', False], ['weakref', False], ['webbrowser', False], ['websocket', True], ['werkzeug', True], ['wheel', True], ['wrapt', True], ['wsgiref', True], ['xdrlib', False], ['xgboost', True], ['xml', True], ['xmlrpc', True], ['xxlimited', False], ['yaml', True], ['zipapp', False], ['zipfile', False], ['zipp', False], ['zlib', False]], 'python_version': '3.7', 'lithops_version': '2.6.1.dev0', 'runtime_timeout': 1200}
job.execution_timeout 21600
after:
job.execution_timeout 1195
job.runtime_timeout 1200
2022-07-28 15:27:52,931 [INFO] invokers.py:175 -- ExecutorID 1006d1-0 | JobID A000 - Starting function invocation: my_function() - Total: 1 activations
2022-07-28 15:27:53,885 [INFO] invokers.py:209 -- ExecutorID 1006d1-0 | JobID A000 - View execution logs at /private/var/folders/w4/gp4gwvjn0413dtbr8y8m02bm0000gn/T/lithops/logs/1006d1-0-A000.log
2022-07-28 15:27:53,885 [INFO] wait.py:97 -- ExecutorID 1006d1-0 - Getting results from 1 function activations

2022-07-28 15:47:56,218 [WARNING] future.py:243 -- ExecutorID 1006d1-0 | JobID A000 - There was an exception - Activation ID: b9211cff9af9
2022-07-28 15:47:56,218 [WARNING] future.py:247 -- Exception: TimeoutError - Function exceeded maximum time of 1195 seconds and was killed

Process finished with exit code 1

Also, I checked code engine dashboard, I can see job timeout is 21600. This seems very strange. Can you suggest what is wrong?

JosepSampe commented 2 years ago

@ShuxinLin Unfortunately I'm not able to replicate your issue:

2022-07-30 02:51:42,101 [INFO] config.py:131 -- Lithops v2.6.1.dev0
2022-07-30 02:51:42,428 [INFO] ibm_cos.py:98 -- IBM COS client created - Region: us-east
2022-07-30 02:51:42,431 [INFO] code_engine.py:131 -- IBM Code Engine client created - Region: eu-de
2022-07-30 02:51:42,431 [INFO] invokers.py:108 -- ExecutorID c1019c-0 | JobID A000 - Selected Runtime: docker.io/jsampe/lithops-ce-default-v39:latest - 256MB
runtime_meta {'preinstalls': [['__future__', False], ['_aix_support', False], ['_asyncio', False], ['_bisect', False], ['_blake2', False], ['_bootlocale', False], ['_bootsubprocess', False], ['_bz2', False], ['_codecs_cn', False], ['_codecs_hk', False], ['_codecs_iso2022', False], ['_codecs_jp', False], ['_codecs_kr', False], ['_codecs_tw', False], ['_collections_abc', False], ['_compat_pickle', False], ['_compression', False], ['_contextvars', False], ['_crypt', False], ['_csv', False], ['_ctypes', False], ['_ctypes_test', False], ['_curses', False], ['_curses_panel', False], ['_datetime', False], ['_decimal', False], ['_distutils_hack', True], ['_elementtree', False], ['_gdbm', False], ['_hashlib', False], ['_heapq', False], ['_json', False], ['_lsprof', False], ['_lzma', False], ['_markupbase', False], ['_md5', False], ['_multibytecodec', False], ['_multiprocessing', False], ['_opcode', False], ['_osx_support', False], ['_pickle', False], ['_posixshmem', False], ['_posixsubprocess', False], ['_py_abc', False], ['_pydecimal', False], ['_pyio', False], ['_queue', False], ['_random', False], ['_sha1', False], ['_sha256', False], ['_sha3', False], ['_sha512', False], ['_sitebuiltins', False], ['_socket', False], ['_sqlite3', False], ['_ssl', False], ['_statistics', False], ['_strptime', False], ['_struct', False], ['_sysconfigdata__linux_x86_64-linux-gnu', False], ['_testbuffer', False], ['_testcapi', False], ['_testimportmultiple', False], ['_testinternalcapi', False], ['_testmultiphase', False], ['_threading_local', False], ['_tkinter', False], ['_uuid', False], ['_weakrefset', False], ['_xxsubinterpreters', False], ['_xxtestfuzz', False], ['_yaml', True], ['_zoneinfo', False], ['abc', False], ['aifc', False], ['antigravity', False], ['argparse', False], ['array', False], ['ast', False], ['async_timeout', True], ['asynchat', False], ['asyncio', True], ['asyncore', False], ['audioop', False], ['base64', False], ['bdb', False], ['binascii', False], ['binhex', False], ['bisect', False], ['bz2', False], ['cProfile', False], ['cachetools', True], ['calendar', False], ['certifi', True], ['cgi', False], ['cgitb', False], ['charset_normalizer', True], ['chunk', False], ['click', True], ['cloudpickle', True], ['cmath', False], ['cmd', False], ['code', False], ['codecs', False], ['codeop', False], ['collections', True], ['colorsys', False], ['compileall', False], ['concurrent', True], ['configparser', False], ['contextlib', False], ['contextvars', False], ['copy', False], ['copyreg', False], ['crypt', False], ['csv', False], ['ctypes', True], ['curses', True], ['dataclasses', False], ['datetime', False], ['dateutil', True], ['dbm', True], ['decimal', False], ['deprecated', True], ['difflib', False], ['dis', False], ['distutils', True], ['doctest', False], ['email', True], ['encodings', True], ['ensurepip', True], ['enum', False], ['fcntl', False], ['filecmp', False], ['fileinput', False], ['flask', True], ['fnmatch', False], ['formatter', False], ['fractions', False], ['ftplib', False], ['functools', False], ['genericpath', False], ['getopt', False], ['getpass', False], ['gettext', False], ['gevent', True], ['glob', False], ['graphlib', False], ['greenlet', True], ['grp', False], ['gunicorn', True], ['gzip', False], ['hashlib', False], ['heapq', False], ['hmac', False], ['html', True], ['http', True], ['ibm_boto3', True], ['ibm_botocore', True], ['ibm_s3transfer', True], ['idlelib', True], ['idna', True], ['imaplib', False], ['imghdr', False], ['imp', False], ['importlib', True], ['importlib_metadata', True], ['inspect', False], ['io', False], ['ipaddress', False], ['itsdangerous', True], ['jinja2', True], ['jmespath', True], ['json', True], ['keyword', False], ['kubernetes', True], ['lib2to3', True], ['linecache', False], ['lithops', True], ['lithopsentry', False], ['locale', False], ['logging', True], ['lzma', False], ['mailbox', False], ['mailcap', False], ['markupsafe', True], ['math', False], ['mimetypes', False], ['mmap', False], ['modulefinder', False], ['multiprocessing', True], ['netrc', False], ['nis', False], ['nntplib', False], ['ntpath', False], ['nturl2path', False], ['numbers', False], ['numpy', True], ['oauthlib', True], ['opcode', False], ['operator', False], ['optparse', False], ['os', False], ['ossaudiodev', False], ['packaging', True], ['parser', False], ['pathlib', False], ['pdb', False], ['pickle', False], ['pickletools', False], ['pika', True], ['pip', True], ['pipes', False], ['pkg_resources', True], ['pkgutil', False], ['platform', False], ['plistlib', False], ['poplib', False], ['posixpath', False], ['pprint', False], ['profile', False], ['ps_mem', False], ['pstats', False], ['pty', False], ['py_compile', False], ['pyasn1', True], ['pyasn1_modules', True], ['pyclbr', False], ['pydoc', False], ['pydoc_data', True], ['pyexpat', False], ['pyparsing', True], ['queue', False], ['quopri', False], ['random', False], ['re', False], ['readline', False], ['redis', True], ['reprlib', False], ['requests', True], ['requests_oauthlib', True], ['resource', False], ['rlcompleter', False], ['rsa', True], ['runpy', False], ['sched', False], ['secrets', False], ['select', False], ['selectors', False], ['setuptools', True], ['shelve', False], ['shlex', False], ['shutil', False], ['signal', False], ['site', False], ['six', False], ['smtpd', False], ['smtplib', False], ['sndhdr', False], ['socket', False], ['socketserver', False], ['spwd', False], ['sqlite3', True], ['sre_compile', False], ['sre_constants', False], ['sre_parse', False], ['ssl', False], ['stat', False], ['statistics', False], ['string', False], ['stringprep', False], ['struct', False], ['subprocess', False], ['sunau', False], ['symbol', False], ['symtable', False], ['sysconfig', False], ['syslog', False], ['tabnanny', False], ['tarfile', False], ['tblib', True], ['telnetlib', False], ['tempfile', False], ['termios', False], ['textwrap', False], ['this', False], ['threading', False], ['timeit', False], ['tkinter', True], ['token', False], ['tokenize', False], ['trace', False], ['traceback', False], ['tracemalloc', False], ['tty', False], ['turtle', False], ['turtledemo', True], ['types', False], ['typing', False], ['unicodedata', False], ['unittest', True], ['urllib', True], ['urllib3', True], ['uu', False], ['uuid', False], ['venv', True], ['warnings', False], ['wave', False], ['weakref', False], ['webbrowser', False], ['websocket', True], ['werkzeug', True], ['wheel', True], ['wrapt', True], ['wsgiref', True], ['xdrlib', False], ['xml', True], ['xmlrpc', True], ['xxlimited', False], ['yaml', True], ['zipapp', False], ['zipfile', False], ['zipimport', False], ['zipp', False], ['zlib', False], ['zoneinfo', True]], 'python_version': '3.9', 'lithops_version': '2.6.1.dev0', 'runtime_timeout': 21600}
job.execution_timeout 21600
after:
job.execution_timeout 21595
job.runtime_timeout 21600
2022-07-30 02:51:43,820 [INFO] invokers.py:172 -- ExecutorID c1019c-0 | JobID A000 - Starting function invocation: hello() - Total: 1 activations
2022-07-30 02:51:45,088 [INFO] invokers.py:208 -- ExecutorID c1019c-0 | JobID A000 - View execution logs at /tmp/lithops/logs/c1019c-0-A000.log
2022-07-30 02:51:45,089 [INFO] wait.py:97 -- ExecutorID c1019c-0 - Getting results from 1 function activations
2022-07-30 02:52:00,325 [INFO] executors.py:609 -- ExecutorID c1019c-0 - Cleaning temporary data

Could it be that you have deployed the runtime with 1200 before changing it to 21600? Note that once the runtime is deployed the runtime_timeout cannot change, this means that lithops will get the runtime_timeout you set the first time from the metadata, independently of what you have at that moment in your config. If you want to set a different runtime_timeout, you first have to undeploy the runtime.

Can you run lithops clean -b code_engine -s ibm_cos and try again?

ShuxinLin commented 2 years ago

problem solved. Thanks!