acederberg commented 2 years ago

Thanks for the awesome tool, it has/will save me a lot of time. I was running this on a new pipeline like

cd <my_project>
pipelines

and it says

bitbucket-pipelines.yml; verify the file contains valid YAML

But I can parse the file with python

from yaml import safe_load
with open( 'bitbucket-pipelines.yml', 'r' )  as file : safe_load( file )

and it passes in the bitbucket validator. When I run the pipeline on bitbucket it works. I can provide my yaml, but it will require some amendments as this is internal to my organization and thus I am hesitant. If I figure out what is making the parser upset I will add that to my post here.

It also might be worth considering that I am using the windows subsystem for linux.

Thank you again for any time, it is really appreciated.

EDIT : Debug output

Running pipelines --debug gets

pipelines: file parse error: YAML error: /mnt/c/MVE/.../bitbucket-pipelines.yml; verify the file contains valid YAML
pipelines: version 0.0.62-composer w/ php 7.4.3 (libyaml: n/a)
--------
class....: Ktomk\Pipelines\File\ParseException
message..: file parse error: YAML error: /mnt/c/.../<my_project>/bitbucket-pipelines.yml; verify the file contains valid YAML
code.....: 2
file.....: /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/File/File.php
line.....: 53
backtrace:
#0 /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/Utility/App.php(135): Ktomk\Pipelines\File\File::createFromFile()
#1 /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/Utility/ExceptionHandler.php(50): Ktomk\Pipelines\Utility\App->run()
#2 /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/Utility/ExceptionHandler.php(65): Ktomk\Pipelines\Utility\ExceptionHandler->handle()
#3 /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/Utility/App.php(90): Ktomk\Pipelines\Utility\ExceptionHandler->handleStatus()
#4 /home/adr1an/.config/composer/vendor/ktomk/pipelines/bin/pipelines(37): Ktomk\Pipelines\Utility\App->main()
#5 /home/adr1an/.config/composer/vendor/bin/pipelines(112): include('/home/adr1an/.c...')
#6 {main}
--------

EDIT

It turns out that the parser does not like anchors and this is the root of the problem.

ktomk commented 2 years ago

@acederberg there can be subtle differences with YAML parsing across platforms. IIRC if you compare w/ python, consider to add the PHP YAML extension if possible. There is a fallback mechanism and if one parser fails, the other is tried. With PHP ext-yaml, there is a second chance.

For the concrete problem if you can add a (reduced is fine, too) bitbucket-pipelines.yml that triggers this issue would be good so I can tell more.

acederberg commented 2 years ago

@ktomk Thanks! I (think I) figured it out. I'm using some YAML anchors and that is causing the issue. I rendered my YAML with python and dumped it into a new pipeline YAML and it works ( it uses all anchors besides the <<: anchor when rendered, so I think that it is the problem ). Would it be possible to add this? IDK anything about PHP hardly, so it's hard to decide where to start.

For example

image : python:3.10

options : { docker : true }

definitions :
  services :
    mongodb :
      environment :
        MONGO_INITDB_ROOT_USERNAME : adrian
        MONGO_INITDB_ROOT_PASSWORD : somepassword
        MONGO_INITDB_DATABASE : tests
      image : mongo
  steps :
    - step : &create_config
        name : "Create '.env'."
        script :
          - export ENV_PATH="$PWD/.env"
          - touch $ENV_PATH
          - echo "mongodb_host='brain_rtus_db'" >> $ENV_PATH
          - echo "mongodb_port=27017" >> $ENV_PATH
          - echo "mongodb_user='adrian'" >> $ENV_PATH
          - echo "mongodb_password='somepassword'" >> $ENV_PATH
          - echo "mongodb_use_atlas='False'" >> $ENV_PATH
          - echo "uvicorn_port=8001" >> $ENV_PATH
          - echo "uvicorn_host='localhost'" >> $ENV_PATH
          - echo "uvicorn_reload='True'" >> $ENV_PATH
          - realpath $ENV_PATH
        artifacts :
          - .env

pipelines :
  default :
    - step :
       <<: *create_config

is invalid. But when I render it with python to get different anchors :

definitions:
  services:
    mongodb:
      environment:
        MONGO_INITDB_DATABASE: tests
        MONGO_INITDB_ROOT_PASSWORD: somepassword
        MONGO_INITDB_ROOT_USERNAME: adrian
      image: mongo
  steps:
  - step:
      artifacts: &id001
      - .env
      name: Create '.env'.
      script: &id002
      - export ENV_PATH="$PWD/.env"
      - touch $ENV_PATH
      - echo "mongodb_host='brain_rtus_db'" >> $ENV_PATH
      - echo "mongodb_port=27017" >> $ENV_PATH
      - echo "mongodb_user='adrian'" >> $ENV_PATH
      - echo "mongodb_password='somepassword'" >> $ENV_PATH
      - echo "mongodb_use_atlas='False'" >> $ENV_PATH
      - echo "uvicorn_port=8001" >> $ENV_PATH
      - echo "uvicorn_host='localhost'" >> $ENV_PATH
      - echo "uvicorn_reload='True'" >> $ENV_PATH
      - realpath $ENV_PATH
image: python:3.10
options:
  docker: true
pipelines :
  default :
  - step:
      artifacts: *id001
      name: Create '.env'.
      script: *id002

it is valid.

ktomk commented 2 years ago

thanks for looking further into it @acederberg

I (think I) figured it out. I'm using some YAML anchors ...

Okay, I can see that. And you got me for a moment because this YAML merge keys can be problematic (in general). I've written a summary of these in YAML Anchor, Aliases and Merge Keys, see especially the section Caveat: Support of Merge Keys.

However, and thanks to your example YAML, I had no problems to use it with pipelines (pipelines: version 0.0.62+6-g3dc9e6a+dirty w/ php 7.4.28 (libyaml: 2.1.0)) and it is not because I have the PHP yaml extension as I have disabled that YAML parser internally for this test.

Toggle me!

~~~ $ bin/pipelines --file test/data/yml/merge-key.yml --step-script --debug # this /bin/sh script is generated from a pipeline script set -e printf '\n' printf '\035+ %s\n' 'export ENV_PATH="$PWD/.env"' export ENV_PATH="$PWD/.env" printf '\n' printf '\035+ %s\n' 'touch $ENV_PATH' touch $ENV_PATH printf '\n' printf '\035+ %s\n' 'echo "mongodb_host='\'brain_rtus_db\''" >> $ENV_PATH' echo "mongodb_host='brain_rtus_db'" >> $ENV_PATH printf '\n' printf '\035+ %s\n' 'echo "mongodb_port=27017" >> $ENV_PATH' echo "mongodb_port=27017" >> $ENV_PATH printf '\n' printf '\035+ %s\n' 'echo "mongodb_user='\'adrian\''" >> $ENV_PATH' echo "mongodb_user='adrian'" >> $ENV_PATH printf '\n' printf '\035+ %s\n' 'echo "mongodb_password='\'somepassword\''" >> $ENV_PATH' echo "mongodb_password='somepassword'" >> $ENV_PATH printf '\n' printf '\035+ %s\n' 'echo "mongodb_use_atlas='\'False\''" >> $ENV_PATH' echo "mongodb_use_atlas='False'" >> $ENV_PATH printf '\n' printf '\035+ %s\n' 'echo "uvicorn_port=8001" >> $ENV_PATH' echo "uvicorn_port=8001" >> $ENV_PATH printf '\n' printf '\035+ %s\n' 'echo "uvicorn_host='\'localhost\''" >> $ENV_PATH' echo "uvicorn_host='localhost'" >> $ENV_PATH printf '\n' printf '\035+ %s\n' 'echo "uvicorn_reload='\'True\''" >> $ENV_PATH' echo "uvicorn_reload='True'" >> $ENV_PATH printf '\n' printf '\035+ %s\n' 'realpath $ENV_PATH' realpath $ENV_PATH pipelines: version 0.0.62+6-g3dc9e6a+dirty w/ php 7.4.28 (libyaml: 2.1.0) -------- class....: Ktomk\Pipelines\Utility\StatusException message..: code.....: 0 file.....: .../pipelines/src/Utility/StatusException.php line.....: 41 backtrace: #0 .../src/Utility/StepScriptOption.php(111): Ktomk\Pipelines\Utility\StatusException::ok() #1 .../src/Utility/App.php(141): Ktomk\Pipelines\Utility\StepScriptOption->run() #2 .../src/Utility/ExceptionHandler.php(50): Ktomk\Pipelines\Utility\App->run() #3 .../src/Utility/ExceptionHandler.php(65): Ktomk\Pipelines\Utility\ExceptionHandler->handle() #4 .../src/Utility/App.php(90): Ktomk\Pipelines\Utility\ExceptionHandler->handleStatus() #5 .../bin/pipelines(38): Ktomk\Pipelines\Utility\App->main() #6 {main} -------- ~~~

So merge keys might be related but also are supported (in pipelines(1)). This is interesting, as it does not work on your end.

Perhaps there is something different in the file triggering the issue?

/E: Test changes here: https://github.com/ktomk/pipelines/commit/497d5050442b8742e3fd32627d273211c39cc4f0 /E: An on this branch: https://github.com/ktomk/pipelines/tree/test-issue-14

acederberg commented 2 years ago

Hi again @ktomk. Looking into it still and it appears that I am entirely wrong about anchors. I completely deanchored another yaml and it does not work, I am free to share this yaml. here it is :

definitions:
  services:
    mysql:
      image: mysql
      variables:
        MYSQL_DATABASE: mve_brain_sqlalchemy_tests
        MYSQL_PASSWORD: somepassword
        MYSQL_RANDOM_ROOT_PASSWORD: 'yes'
        MYSQL_USER: adrian
  steps:
  - step:
      artifacts:
      - .env.test
      - fetchers/.weather.env
      name: Create .env.test for testing since this file should never be commited.
      script:
      - export ENV_TEST_PATH=".env.test"
      - touch $ENV_TEST_PATH
      - echo 'mysql_database="mve_brain_sqlalchemy_tests"' >> $ENV_TEST_PATH
      - echo 'mysql_host=127.0.0.1' >> $ENV_TEST_PATH
      - echo 'mysql_port=3306' >> $ENV_TEST_PATH
      - echo 'mysql_username="adrian"' >> $ENV_TEST_PATH
      - echo 'mysql_password="somepassword"' >> $ENV_TEST_PATH
      - echo 'mysql_drivername="mysql+asyncmy"' >> $ENV_TEST_PATH
      - echo 'uvicorn_host="0.0.0.0"' >> $ENV_TEST_PATH
      - echo 'uvicorn_port=8000' >> $ENV_TEST_PATH
      - export ENV_TEST_FETCHERS=./api/fetchers/.fetchers.env
      - touch $ENV_TEST_FETCHERS
      - echo "weather_key=$WEATHER_KEY" >> $ENV_TEST_FETCHERS
image: python:3.10
options:
  docker: true
pipelines:
  custom:
    build_pipeline_service:
    - step:
        artifacts:
        - .env.test
        - fetchers/.weather.env
        name: Create .env.test for testing since this file should never be commited.
        script:
        - export ENV_TEST_PATH=".env.test"
        - touch $ENV_TEST_PATH
        - echo 'mysql_database="mve_brain_sqlalchemy_tests"' >> $ENV_TEST_PATH
        - echo 'mysql_host=127.0.0.1' >> $ENV_TEST_PATH
        - echo 'mysql_port=3306' >> $ENV_TEST_PATH
        - echo 'mysql_username="adrian"' >> $ENV_TEST_PATH
        - echo 'mysql_password="somepassword"' >> $ENV_TEST_PATH
        - echo 'mysql_drivername="mysql+asyncmy"' >> $ENV_TEST_PATH
        - echo 'uvicorn_host="0.0.0.0"' >> $ENV_TEST_PATH
        - echo 'uvicorn_port=8000' >> $ENV_TEST_PATH
        - export ENV_TEST_FETCHERS=./api/fetchers/.fetchers.env
        - touch $ENV_TEST_FETCHERS
        - echo "weather_key=$WEATHER_KEY" >> $ENV_TEST_FETCHERS
    - step:
        name: "Building an api image to be used in other pipelines using development\
          \ container multistage build. \nThis includes the .env file constructed\
          \ here.\n"
        script:
        - cp .env.test .env
        - export IMAGE_NAME="$DOCKERHUB_USERNAME/$DOCKERHUB_BRAIN_REPOSITORY"
        - export IMAGE_PIPELINE_SERVICE="$IMAGE_NAME:api_pipeline_service_$BITBUCKET_COMMIT"
        - docker login --username $DOCKERHUB_USERNAME --password $DOCKERHUB_PASSWORD
        - docker build -t "$IMAGE_PIPELINE_SERVICE" -f "Dockerfile" .
        - docker run --name test2 --detach "$IMAGE_PIPELINE_SERVICE"
        - sleep 15
        - docker stop test2
        - docker push $IMAGE_NAME
  default:
  - step:
      caches:
      - pip
      name: Install all dependencies.
      script:
      - pip install -r requirements.txt
      - pip install -r requirements.dev.txt
  - step:
      artifacts:
      - .env.test
      - fetchers/.weather.env
      name: Create .env.test for testing since this file should never be commited.
      script:
      - export ENV_TEST_PATH=".env.test"
      - touch $ENV_TEST_PATH
      - echo 'mysql_database="mve_brain_sqlalchemy_tests"' >> $ENV_TEST_PATH
      - echo 'mysql_host=127.0.0.1' >> $ENV_TEST_PATH
      - echo 'mysql_port=3306' >> $ENV_TEST_PATH
      - echo 'mysql_username="adrian"' >> $ENV_TEST_PATH
      - echo 'mysql_password="somepassword"' >> $ENV_TEST_PATH
      - echo 'mysql_drivername="mysql+asyncmy"' >> $ENV_TEST_PATH
      - echo 'uvicorn_host="0.0.0.0"' >> $ENV_TEST_PATH
      - echo 'uvicorn_port=8000' >> $ENV_TEST_PATH
      - export ENV_TEST_FETCHERS=./api/fetchers/.fetchers.env
      - touch $ENV_TEST_FETCHERS
      - echo "weather_key=$WEATHER_KEY" >> $ENV_TEST_FETCHERS
  - step:
      caches:
      - pip
      name: Test
      script:
      - pip install -r requirements.txt
      - pip install -r requirements.dev.txt
      - python -m pytest .
      services:
      - mysql
  - step:
      caches:
      - pip
      name: Building prod and fetchers docker images (without an enironment files)
      script:
      - export IMAGE_NAME="$DOCKERHUB_USERNAME/$DOCKERHUB_BRAIN_REPOSITORY"
      - export IMAGE_API="$IMAGE_NAME:api_$BITBUCKET_COMMIT"
      - export IMAGE_FETCHERS="$IMAGE_NAME:fetchers_$BITBUCKET_COMMIT"
      - docker login --username $DOCKERHUB_USERNAME --password $DOCKERHUB_PASSWORD
      - docker build -t "$IMAGE_API" -f "Dockerfile.prod" --target "prod" .
      - docker build -t "$IMAGE_FETCHERS" -f "Dockerfile.prod" --target "fetcher_runner"
        .
      - docker run --name test1 --detach "$IMAGE_API"
      - sleep 30
      - docker stop test1
      - docker push $IMAGE_NAME

ktomk commented 2 years ago

With the Sf2Yaml it works on my end.

With the LibYaml it does not and I could find two "culprits":

The double quoted string (line 56, name) with the line continuations looks not supported
The spurious dot (line 117) looks not supported

If I remove both, it works with the libyaml parser, too.

Wondering a bit why it's not working on your end as IIRC the Sf2Yaml parser is taken on your end as you don't have LibYaml. @acederberg

acederberg commented 2 years ago

Here's the verbose output prior to those changes mentioned above :

❯ pipelines --file bitbucket-pipelines-deanchored.yml --debug --verbose
info: project directory is '/mnt/c/MVE/mve-brain-services/mve-brain-api'
info: pipelines file is '/mnt/c/MVE/mve-brain-services/mve-brain-api/bitbucket-pipelines-deanchored.yml'
pipelines: file parse error: YAML error: /mnt/c/MVE/mve-brain-services/mve-brain-api/bitbucket-pipelines-deanchored.yml; verify the file contains valid YAML
pipelines: version 0.0.62-composer w/ php 7.4.3 (libyaml: n/a)
--------
class....: Ktomk\Pipelines\File\ParseException
message..: file parse error: YAML error: /mnt/c/MVE/mve-brain-services/mve-brain-api/bitbucket-pipelines-deanchored.yml; verify the file contains valid YAML
code.....: 2
file.....: /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/File/File.php
line.....: 53
backtrace:
#0 /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/Utility/App.php(135): Ktomk\Pipelines\File\File::createFromFile()
#1 /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/Utility/ExceptionHandler.php(50): Ktomk\Pipelines\Utility\App->run()
#2 /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/Utility/ExceptionHandler.php(65): Ktomk\Pipelines\Utility\ExceptionHandler->handle()
#3 /home/adr1an/.config/composer/vendor/ktomk/pipelines/src/Utility/App.php(90): Ktomk\Pipelines\Utility\ExceptionHandler->handleStatus()
#4 /home/adr1an/.config/composer/vendor/ktomk/pipelines/bin/pipelines(37): Ktomk\Pipelines\Utility\App->main()
#5 /home/adr1an/.config/composer/vendor/bin/pipelines(112): include('/home/adr1an/.c...')
#6 {main}
--------

I got it to run after fixing those, so it looks like some whacky stuff is happening with pyyaml causing it to dump the YAML with those errors. I am still ( and now to a greater degree than before ) confused as there is still something in my other yaml that will make it fail after it's rendered. IDK what I am doing wrong elsewhere, but it must be an error in my yaml and not the parser itself.

Is there a way to get better feedback on the YAML errors from pipelines? e.g. "bad syntax on line n column k"

ktomk commented 2 years ago

Well the YAML parser in pipelines most likely isn't golden. I had issues earlier and from where its coming from it was always with some fallback. I'll tweak the handling based on the feedback here probably. Thinking about putting the Sf2Yaml parser upfront and use the LibYaml one as fallback (currently the other way round, but what I learned this is good for a change).

Is there a way to get better feedback on the YAML errors from pipelines? e.g. "bad syntax on line n column k"

Yes, that would be good to at least have some pointers. I'll patch something in, it should provide more info at least in some cases. It's not entirely structural like line and column AFAIK.

ktomk commented 2 years ago

Okay, I made a mistake in testing as I now see:

Malformed inline YAML string ("Building an api image to be used in other pipelines using development) at line 56 (near "name: "Building an api image to be used in other pipelines using development\").

This is from the Sf2Yaml parser. The LibYaml parser is fine. Looks like I mixed it up. That should explain why I was puzzled earlier. I'll put error messages in. It will be a bit text-walling but better than current.

ktomk commented 2 years ago

Version 0.0.63 has been released showing more detailed error messages on YAML file parse errors, Link to full change-log..

acederberg commented 2 years ago

@ktomk Thank you! For the reference of anybody who needs help in the future who is unfamiliar with php package managers, update by editing the version number in ~/.config/composer/composer.json and then run composer global update ktomk/pipelines.

ktomk commented 2 years ago

@acederberg: Yes, if you install with composer global, you can update within the stable channel even without editing the composer.json by hand:

composer global require ktomk/pipelines:^0.0

Takes the latest 0.0.x version (as require also updates).

It is using require here so it works with however you have set it up originally.

As it also works from the source version, it should be possible as well to install non-tagged revisions with composer:

$ composer global require ktomk/pipelines:dev-test-issue-14
...
$ pipelines --version
pipelines version dev-test-issue-14-composer

How did you learn about installing this utility?

Just curious and I may think about improving the docs about it.

acederberg commented 2 years ago

@ktomk Hi Just saw your last comment. I found it by searching 'run bitbucket pipelines locally' out of frustration with the workflow of debugging pipelines. I found your kb which was the best resource I could find on the topic.

ktomk / pipelines

(Some) Anchors Do Not Work - Raises Invalid YAML #14

EDIT : Debug output

EDIT