search

terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

ERR_INVALID_ARG_TYPE when submitting Teraslice Job #3692

Open godber opened 1 month ago

godber commented 1 month ago

When trying to register the following job:

{
    "name": "ts-tmp1-test",
    "lifecycle": "persistent",
    "workers": 1,
    "assets": [
        "standard",
        "elasticsearch"
    ],
    "operations": [
        {
            "_op": "data_generator",
            "size": 10000
        },
        {
            "_op": "elasticsearch_bulk",
            "index": "testjob1",
            "size": 10000
        }
    ]
}

we get the following error

earl tjm register tmp1 ts-tmp1-test.json
Error Failure to get assets, caused by TypeError [ERR_INVALID_ARG_TYPE]: The first argument must be of type string or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object. Received undefined
    at new NodeError (node:internal/errors:405:5)
  1 {
    at Function.from (node:buffer:325:9)
    at file:///app/source/packages/teraslice/dist/src/lib/workers/assets/loader.js:54:31
    at async Promise.all (index 1)
    at async AssetLoader.load (file:///app/source/packages/teraslice/dist/src/lib/workers/assets/loader.js:41:27)
    at async file:///app/source/packages/teraslice/dist/src/lib/workers/assets/loader-executable.js:18:34
If running out of memory, try consider increasing the memory allocation for the process by adding/modifying the "memory_execution_controller" or "resources_limits_memory" (for workers) field in the job file. registering job ts-tmp1-test on https://ts-tmp1

Here are the latest assets

curl -Ss https://ts-tmp1/txt/assets
name               version        id                                        _created                  description                     node_version  platform  arch
-----------------  -------------  ----------------------------------------  ------------------------  ------------------------------  ------------  --------  ----
file               3.0.0          6cdfa4cad29caaad0dbe746150c224197a68550d  2024-07-23T17:10:01.460Z  A set of processors for workin  18
standard           1.0.0          368c7ed1c48c5021c04d1948dde27c2914bdea94  2024-07-23T17:09:00.675Z  Teraslice standard processor a  18
kafka              5.0.0          26e9c484cf30fd1a5dba5f24f291c454e1fdf166  2024-07-23T17:08:18.995Z  Kafka reader and writer suppor  18
elasticsearch      4.0.0          09cf49745ddecf6f65338029b7310059b2f2e4e5  2024-07-23T17:05:17.275Z                                  18
godber commented 1 month ago

This error occurs when Teraslice is configured with the removed terafoundation.asset_storage_connection_type instead of teraslice.asset_storage_connection_type. The key was removed in the 2.0.0 release. I am changing this from bug to enhancement. We should improve the messaging around this error. THoughts: