search

turbot / flowpipe

Flowpipe is a cloud scripting engine. Automation and workflow to connect your clouds to the people, systems and data that matters.
https://flowpipe.io
GNU Affero General Public License v3.0
385 stars 16 forks source link

Error is not shown when running GCP mod pipeline locally without `application_credentials_path` variable set #414

Closed cbruno10 closed 12 months ago

cbruno10 commented 12 months ago

From the Add-new-pipeline-and-code-clean-up branch, if I run any pipeline without setting the vars, I get an error but don't see the error on the terminal, but can see it in my logs:

cbruno@M1P gcp % flowpipe pipeline run list_pubsub_topics
[Execution] exec_clnoqd7e58cht2m8qaf0
[list_pubsub_topics] Starting: pexec_clnoqd7e58cht2m8qafg
[list_pubsub_topics] Failed with 1 error(s) 1ms
cbruno@M1P gcp % cat .flowpipe/store/exec_clnoqd7e58cht2m8qaf0.jsonl
{"level":"info","ts":"2023-12-05T20:51:00Z","caller":"command","msg":"es","event_type":"command.pipeline_queue","payload":{"event":{"execution_id":"exec_clnoqd7e58cht2m8qaf0","created_at":"2023-12-05T20:51:00.869268Z"},"name":"gcp.pipeline.list_pubsub_topics","args":null,"pipeline_execution_id":"pexec_clnoqd7e58cht2m8qafg"}}
{"level":"info","ts":"2023-12-05T20:51:00Z","caller":"command","msg":"es","event_type":"handler.pipeline_queued","payload":{"event":{"execution_id":"exec_clnoqd7e58cht2m8qaf0","created_at":"2023-12-05T20:51:00.871304Z"},"name":"gcp.pipeline.list_pubsub_topics","args":null,"pipeline_execution_id":"pexec_clnoqd7e58cht2m8qafg"}}
{"level":"info","ts":"2023-12-05T20:51:00Z","caller":"command","msg":"es","event_type":"command.pipeline_load","payload":{"event":{"execution_id":"exec_clnoqd7e58cht2m8qaf0","created_at":"2023-12-05T20:51:00.871961Z"},"pipeline_execution_id":"pexec_clnoqd7e58cht2m8qafg"}}
{"level":"info","ts":"2023-12-05T20:51:00Z","caller":"command","msg":"es","event_type":"handler.pipeline_loaded","payload":{"event":{"execution_id":"exec_clnoqd7e58cht2m8qaf0","created_at":"2023-12-05T20:51:00.873006Z"},"pipeline_execution_id":"pexec_clnoqd7e58cht2m8qafg","pipeline":{"name":"","pipeline_name":"gcp.pipeline.list_pubsub_topics","steps":[{"name":"list_pubsub_topics","step_type":"container","pipeline_name":"gcp.pipeline.list_pubsub_topics","image":"my-gcloud-image-latest","source":null,"cmd":["pubsub","topics","list"],"env":null,"entrypoint":null,"timeout":null,"cpu_shares":null,"memory":null,"memory_reservation":null,"memory_swap":null,"memory_swappiness":null,"read_only":null,"user":null,"workdir":null}],"outputs":[{"name":"stdout","description":"The JSON output from the GCP CLI.","depends_on":["container.list_pubsub_topics"]},{"name":"stderr","description":"The error output from the GCP CLI.","depends_on":["container.list_pubsub_topics"]}],"params":{"application_credentials_path":{"name":"application_credentials_path","description":"The GCP application credentials file path."},"project_id":{"name":"project_id","description":"The GCP project ID."}}}}}
{"level":"info","ts":"2023-12-05T20:51:00Z","caller":"command","msg":"es","event_type":"command.pipeline_start","payload":{"event":{"execution_id":"exec_clnoqd7e58cht2m8qaf0","created_at":"2023-12-05T20:51:00.875823Z"},"pipeline_execution_id":"pexec_clnoqd7e58cht2m8qafg"}}
{"level":"info","ts":"2023-12-05T20:51:00Z","caller":"command","msg":"es","event_type":"handler.pipeline_started","payload":{"event":{"execution_id":"exec_clnoqd7e58cht2m8qaf0","created_at":"2023-12-05T20:51:00.875987Z"},"pipeline_execution_id":"pexec_clnoqd7e58cht2m8qafg"}}
{"level":"info","ts":"2023-12-05T20:51:00Z","caller":"command","msg":"es","event_type":"command.pipeline_plan","payload":{"event":{"execution_id":"exec_clnoqd7e58cht2m8qaf0","created_at":"2023-12-05T20:51:00.876129Z"},"pipeline_execution_id":"pexec_clnoqd7e58cht2m8qafg"}}
{"level":"info","ts":"2023-12-05T20:51:00Z","caller":"command","msg":"es","event_type":"handler.pipeline_failed","payload":{"event":{"execution_id":"exec_clnoqd7e58cht2m8qaf0","created_at":"2023-12-05T20:51:00.877206Z"},"pipeline_execution_id":"pexec_clnoqd7e58cht2m8qafg","error":[{"pipeline_execution_id":"pexec_clnoqd7e58cht2m8qafg","step_execution_id":"","pipeline":"gcp.pipeline.list_pubsub_topics","step":"list_pubsub_topics","error":{"instance":"fperr_clnoqd7e58cht2m8qagg","type":"error_internal","title":"Internal Error","status":500,"detail":"list_pubsub_topics: Invalid function argument: Invalid value for \"path\" parameter: failed to read file: read .: is a directory.\n(/Users/cbruno/flowpipe/gcp/pipelines/pubsub/list_pubsub_topic.fp:21,24-58)"}}],"pipeline_output":null}}
graza-io commented 12 months ago

Thanks, I think this one is already on track - I've attached the branch currently working on which has a lot of rework around the pipeline run output.