Indexing of base URI interrupted on unknown error

[jotelha]

The implementation of flask command flask base_uri index BASE_URI will stop iterating when an error other than the predefined ones (like yaml.parser.ParserError, yaml.scanner.ScannerError) arises, https://github.com/jic-dtool/dtool-lookup-server/blob/8b1071b3bae85b0dd29de2923dccb06444456b37/dtool_lookup_server/cli.py#L190-L218,

Today, I found some storage broker error arising within generate_dataset_info causing the indexing process to interrupt with zero exit code,

...
Registered: s3://frct-simdata/6fc50222-5d53-4275-96aa-0dc4b569e365
Registered: s3://frct-simdata/6fcd2417-961c-46a0-840d-8f166fe1de8c
Registered: s3://frct-simdata/6fcdfbe9-654d-40cb-b0c6-6f6edf59472f
Registered: s3://frct-simdata/6fd07674-afca-4191-89b5-f602eb1e77d7
Registered: s3://frct-simdata/6fd19c2b-5f3e-42a2-b421-0a7066e9ae3c
Traceback (most recent call last):
  File "/usr/local/bin/flask", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/flask/cli.py", line 967, in main
    cli.main(args=sys.argv[1:], prog_name="python -m flask" if as_module else None)
  File "/usr/local/lib/python3.9/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/src/dtool-lookup-server/dtool_lookup_server/cli.py", line 236, in index_base_uri
    dataset_info = generate_dataset_info(dataset, base_uri)
  File "/src/dtool-lookup-server/dtool_lookup_server/utils.py", line 197, in generate_dataset_info
    dataset.get_readme_content(),
  File "/usr/local/lib/python3.9/site-packages/dtoolcore/__init__.py", line 455, in get_readme_content
    return self._storage_broker.get_readme_content()
  File "/usr/local/lib/python3.9/site-packages/dtoolcore/storagebroker.py", line 249, in get_readme_content
    return self.get_text(self.get_readme_key())
  File "/usr/local/lib/python3.9/site-packages/dtool_s3/storagebroker.py", line 439, in get_text
    response = self.s3resource.Object(
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/factory.py", line 580, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/action.py", line 88, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 395, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 725, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.
+ exit 0

I will prepare a pull request that catches arbitrary errors.

I will also suggest removing the YAML parsing errors as they are obsolete now.