bootstrapper cannot handle input with wildcards

version 1.3

create a notebook1 that produces several output files, e.g. data/file1.csv, data/file2.csv, ...
create a notebook2 (it doesn't matter what it does because it won't run)
create a pipeline notebook1 -> notebook2, configuring the output files for notebook1 as data/*
run the pipeline on kubeflow pipelines

Result:

processing of notebook1 succeeds and the output files are properly uploaded to the COS bucket
processing of notebook2 fails:

[I 23:08:27.715] 'test_load_viz-1023160715':'Data_Viz' - downloaded Data_Viz-2289c970-b214-418b-bf93-68d880326eb0.tar.gz from bucket: pipeline-artifacts, object: test_load_viz-1023160715/Data_Viz-2289c970-b214-418b-bf93-68d880326eb0.tar.gz (0.042 secs)
Traceback (most recent call last):
  File "bootstrapper.py", line 402, in <module>
    main()
  File "bootstrapper.py", line 393, in main
    file_op.process_dependencies()
  File "bootstrapper.py", line 97, in process_dependencies
    self.get_file_from_object_storage(file.strip())
  File "bootstrapper.py", line 137, in get_file_from_object_storage
    self.cos_client.fget_object(bucket_name=self.cos_bucket,
  File "/usr/local/lib/python3.8/site-packages/minio/api.py", line 719, in fget_object
    stat = self.stat_object(bucket_name, object_name, sse)
  File "/usr/local/lib/python3.8/site-packages/minio/api.py", line 1138, in stat_object
    response = self._url_open('HEAD', bucket_name=bucket_name,
  File "/usr/local/lib/python3.8/site-packages/minio/api.py", line 2017, in _url_open
    raise ResponseError(response,
minio.error.NoSuchKey: NoSuchKey: message: The specified key does not exist.

The corresponding lines of code are

        if inputs:
            input_list = inputs.split(INOUT_SEPARATOR)
            for file in input_list:
                self.get_file_from_object_storage(file.strip()).     <------ FAIL

To figure out which input file couldn't be processed I had to export the pipeline and inspect the generated bootstrapper script:

- name: data-viz
    container:
      args: ['mkdir -p ./jupyter-work-dir/ && cd ./jupyter-work-dir/ && curl -H "Cache-Control:
          no-cache" -L https://raw.githubusercontent.com/elyra-ai/kfp-notebook/v0.13.0/etc/docker-scripts/bootstrapper.py
          --output bootstrapper.py && curl -H "Cache-Control: no-cache" -L https://raw.githubusercontent.com/elyra-ai/kfp-notebook/v0.13.0/etc/requirements-elyra.txt
          --output requirements-elyra.txt && python3 -m pip install  packaging &&
          python3 -m pip freeze > requirements-current.txt && python3 bootstrapper.py
          --cos-endpoint http://devises1.fyre.ibm.com:31323 --cos-bucket pipeline-artifacts
          --cos-directory "test_load_viz" --cos-dependencies-archive "Data_Viz-2289c970-b214-418b-bf93-68d880326eb0.tar.gz"
          --file "XAI/Elyra Build/Data_Viz.ipynb" --inputs "data/bank-additional/*" ']

Input is set to data/bank-additional/*, which seems to cause the failure.

Issues:

when loading declared output files, special processing is required if the input "name" contains wildcards
the error message should identify the key that was used when an attempt was made to download an object from the bucket

elyra-ai / elyra

bootstrapper cannot handle input with wildcards #995