RegEx Syntax issue in roles/vdm/defaults/main.yaml

[Marc-sas]

Hello Experts!

I'm running this code and I seem to have run into a kind of syntax issue. Do you think you can review it?

The error appears to be in 'viya4-deployment/public/roles/vdm/defaults/main.yaml': line 15, column 60, but may be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

V4_CFG_CR_URL: https://cr.sas.com V4_CFG_CR_HOST: "{{ V4_CFG_CR_URL | regex_replace('^https?:\/\/(.*)\/?', '\1') }}" ^ here

[thpang]

More info here would be appreciated. Output/Error message, etc.

[Marc-sas]

Hi I apologize,

I was executing the ansible here: https://github.com/sassoftware/viya4-deployment/blob/main/docs/user/AnsibleUsage.md

malepr@cldlgn04:~/clouddrive/project/aks/viya4-deployment/public$ ansible-playbook \
>   -e BASE_DIR=$HOME/clouddrive/project/aks/viya4-deployment/public \
>   -e CONFIG=$HOME/clouddrive/project/aks/viya4-deployment/public/ansible-vars-iac-azure.yaml \
>   -e TFSTATE=$HOME/clouddrive/project/aks/viya4-iac-azure/terraform.tfstate \
>   -e JUMP_SVR_PRIVATE_KEY=$HOME/.ssh/azure/id_rsa \
>   $HOME/clouddrive/project/aks/viya4-deployment/public/playbooks/playbook.yaml --tags "baseline,viya,cluster-logging,cluster-monitoring,viya-monitoring,install"

And during its execution I received the following error - I believe I have captured it all and this should be sufficient to recreate the problem right?

ERROR! We were unable to read either as JSON nor YAML, these are the errors we got from each:
JSON: Expecting value: line 1 column 1 (char 0)

Syntax Error while loading YAML.
  found unknown escape character

The error appears to be in '/viya4-deployment/public/roles/vdm/defaults/main.yaml': line 15, column 60, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

V4_CFG_CR_URL: https://cr.sas.com
V4_CFG_CR_HOST: "{{ V4_CFG_CR_URL | regex_replace('^https?:\/\/(.*)\/?', '\\1') }}"
                                                           ^ here

What's further is I can copy/paste the roles/vdm/defaults/main.yaml code into yamllint.com and get a syntax error:

[thpang]

You file has a special character in it. It's not been saved as a text file. It probably has a BOM value at the begging of the file. I'd suggest pulling the original file and using vi or nano to edit the file. The actual error is here:

ERROR! We were unable to read either as JSON nor YAML, these are the errors we got from each: JSON: Expecting value: line 1 column 1 (char 0)

Syntax Error while loading YAML. found unknown escape character

[Marc-sas]

Hi Thomas,

I'm really not sure what's going on, but I'm having a real tough time removing the BOMs. I have gone done:

vi main.yaml
:set nobomb
:wq 

Then taken the text and copy/pasted it into yamllint.com and I still get the same issue as above. (Also re-ran the ansible playbook command and got the same error again).

I have also gone to the raw version of the yaml code in the browser - and directly copy/pasted it into notepad++ to view all characters and I see nothing wrong - yet when I copy/paste into yamllint.com and the main.yaml (and run the ansible-playbook) - I receive this same issue.

Also - I have changed the files encoding to ansi and running the ansible-playbook AND trying it with UTF-8 in vi using :write ++enc=ansi and :write ++enc=utf-8 Neither seem to have fixed my problem.

Finally - using the raw form of code from the browser, I have used the cat and stdout methods as a way of writing into the file to try and avoid BOMs and this did not work either. Any suggestions here? Am I going mad?

cat > main.yaml <<-EOF
PROVIDER ...
...
EOF

[Marc-sas]

One more thing - when I manually type out the line below in yamllint - I still get the error: V4_CFG_CR_HOST: "{{ V4_CFG_CR_URL | regex_replace('^https?:\/\/(.*)\/?', '\1' }}"

Are you sure this is related to BOM? I still believe this is syntax related.

[Marc-sas]

Finally - after some random guessing - I seem to have stumbled upon some valid syntax that yamllint helped auto-fill for me: V4_CFG_CR_HOST: "\\\"{{ V4_CFG_CR_URL | regex_replace('^https?:\\/\\/(.*)\\/?', '\\\\\\\\1' }}\""

After running the ansible-playbook, I was able to progress past this issue and encounter a different one (now it seems my cadence version is wrong...)

I highly suggest making the correction to the roles/vdm/defaults/main.yaml so that others do not run into the same issue as me.

[thpang]

@Marc-sas the regex works fine on the current linux systems not sure why you're having so many issues, others are not. Are you trying to run this from an Windows ENV? Asking as Windows is not supported.

[Marc-sas]

Hi Thomas, I'm using the below linux system as the "operator" or control node Linux <HOSTNAME> 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

But this doesn't explain why YAMLlint still found the syntax error.

[thpang]

And you've verified your version dependencies here: https://github.com/sassoftware/viya4-deployment/blob/main/docs/user/Dependencies.md#dependency-versions

[Marc-sas]

Hi Thomas,

As far as I can tell - these requirements are satisfied:

malepr@cldlgn01:~/clouddrive/project/aks/viya4-deployment/public$ python3 --version
Python 3.6.9
malepr@cldlgn01:~/clouddrive/project/aks/viya4-deployment/public$ python --version
Python 2.7.17
malepr@cldlgn01:~/clouddrive/project/aks/viya4-deployment/public$ pip --version
pip 21.1.1 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)
malepr@cldlgn01:~/clouddrive/project/aks/viya4-deployment/public$ kustomize version
{Version:kustomize/v3.7.0 GitCommit:42d1f7b792a80af34828ec4c89af99e1043351a7 BuildDate:2020-07-04T19:15:46Z GoOs:linux GoArch:amd64}
malepr@cldlgn01:~/clouddrive/project/aks/viya4-deployment/public$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.9", GitCommit:"6c90dbd9d6bb1ae8a4c0b0778752be06873e7c55", GitTreeState:"clean", BuildDate:"2021-03-22T23:02:49Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
malepr@cldlgn01:~/clouddrive/project/aks/viya4-deployment/public$ ansible --version
ansible 2.10.8
  config file = /r/ge.unx.sas.com/vol/vol620/u62/malepr/clouddrive/project/aks/viya4-deployment/public/ansible.cfg
  configured module search path = ['/usr/share/ansible', '/r/ge.unx.sas.com/vol/vol620/u62/malepr/clouddrive/project/aks/viya4-deployment/public/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.9 (default, Jan 26 2021, 15:33:00) [GCC 8.4.0]
malepr@cldlgn01:~/clouddrive/project/aks/viya4-deployment/public$ ansible-galaxy --version
ansible-galaxy 2.10.8
  config file = /r/ge.unx.sas.com/vol/vol620/u62/malepr/clouddrive/project/aks/viya4-deployment/public/ansible.cfg
  configured module search path = ['/usr/share/ansible', '/r/ge.unx.sas.com/vol/vol620/u62/malepr/clouddrive/project/aks/viya4-deployment/public/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
  executable location = /usr/local/bin/ansible-galaxy
  python version = 3.6.9 (default, Jan 26 2021, 15:33:00) [GCC 8.4.0]

I didn't check the versions of these:

pip3 | openshift | 0.11.2 -- | -- | -- pip3 | kubernetes | 11.0.0 pip3 | dnspython | 2.1.0 ansible-galaxy | community.kubernetes | 1.2.0

Let me know if these are necessary as well to evaluate the YAML discrepancy. I think the key one is ansible 2.10.8 vs the requirement doc says 2.10.0 which shows you the python version it is using.

[thpang]

@Marc-sas try and clone the repo to a non NFS drive location and run from there.

[thpang]

@Marc-sas any progress?

[Marc-sas]

Hi Thomas, I'm not sure I will be able to get to this today - I will try and run this from a bastion host in Azure on a dedicated file system. To set that up, I'll need some time (primarily troubleshooting) =) I will provide an update when I have done so.

[Marc-sas]

After much deliberation - I was able to set up the default jumpbox set up in the viya4-iac-azure with a public IP, downloaded all necessary required software components and was able to run the code. Again - I ran into the same issue. Here is the code I submitted: jumpuser@malepr-viya4-jump-vm:/opt/viya4/viya4-deployment$ ansible-playbook -e BASE_DIR=/opt/viya4/viya4-deployment -e CONFIG=/opt/viya4/viya4-deployment/ansible-vars-iac-azure.yaml -e TFSTATE=/opt/viya4/viya4-deployment/terraform.tfstate -e JUMP_SVR_PRIVATE_KEY=$HOME/.ssh/id_rsa /opt/viya4/viya4-deployment/playbooks/playbook.yaml --tags "baseline,viya,cluster-logging,cluster-monitoring,viya-monitoring,install"

And here is the error:

ERROR! We were unable to read either as JSON nor YAML, these are the errors we got from each:
JSON: Expecting value: line 1 column 1 (char 0)

Syntax Error while loading YAML.
  found unknown escape character

The error appears to be in '/opt/viya4/viya4-deployment/roles/vdm/defaults/main.yaml': line 15, column 60, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

V4_CFG_CR_URL: https://cr.sas.com
V4_CFG_CR_HOST: "{{ V4_CFG_CR_URL | regex_replace('^https?:\/\/(.*)\/?', '\\1') }}"
                                                           ^ here
We could be wrong, but this one looks like it might be an issue with
missing quotes. Always quote template expression brackets when they
start a value. For instance:

    with_items:
      - {{ foo }}

Should be written as:

    with_items:
      - "{{ foo }}"

[thpang]

How are you installing the files onto the /opt/viya4 locations are you using git clone or cp or tar. I just created a VM in Azure with the same specs as our jump box. Cloned the viya4-deployment repo and had it parse the CONFIG file without errors. So if you can share your steps on setting up your system that would be helpful.

[Marc-sas]

Hi Thomas, Here is the command I used to download the repo: git clone -b main https://github.com/sassoftware/viya4-deployment.git

At this point - its probably best if I walk through and rebuild it so you can see it for yourself. I don't have a clue why this is happening.

[thpang]

So when you did your clean setup did you edit the ansible vars file locally on the linux os or did you do the VSCode route again on Windows?

[Marc-sas]

I am editing through VS Code - but keep in mind, this is terminal display. I'm not opening the files in VSCode. Do you think scheduling a call will help?

I highly doubt there's any BOMs here

From: Thomas S. Pangborn @.> Sent: Friday, May 14, 2021 11:14 AM To: sassoftware/viya4-deployment @.> Cc: Marc Leprince @.>; Mention @.> Subject: Re: [sassoftware/viya4-deployment] RegEx Syntax issue in roles/vdm/defaults/main.yaml (#87)

EXTERNAL

So when you did your clean setup did you edit the ansible vars file locally on the linux os or did you do the VSCode route again on Windows?

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsassoftware%2Fviya4-deployment%2Fissues%2F87%23issuecomment-841305938&data=04%7C01%7Cmarc.leprince%40sas.com%7Cabefed3dc86d49fc2ac808d916eadb0d%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C637566020232722115%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZzG8XytwozH89qAQRq%2Fj%2BZnptSm15EKtKlSrQhNnMl8%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAT7FTQCMEFP4K7I36EKRB4TTNU42HANCNFSM44HMQUWA&data=04%7C01%7Cmarc.leprince%40sas.com%7Cabefed3dc86d49fc2ac808d916eadb0d%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C637566020232732108%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Ihnm9xuHV0UeKNgehk7SA%2FrtfER8hQKWPxmZ%2FTBOrgc%3D&reserved=0.

[thpang]

Can't hurt. I am free today between 1:00pm up to 3:00pm. If that does not work we'd need to chat next week.

[Marc-sas]

Let's chat next week. I've got a client meeting in Little Rock next Monday/Tuesday.

I'll rebuild the whole thing from scratch.

Cheers, Marc

From: Thomas S. Pangborn @.> Sent: Friday, May 14, 2021 11:33 AM To: sassoftware/viya4-deployment @.> Cc: Marc Leprince @.>; Mention @.> Subject: Re: [sassoftware/viya4-deployment] RegEx Syntax issue in roles/vdm/defaults/main.yaml (#87)

EXTERNAL

Can't hurt. I am free today between 1:00pm up to 3:00pm. If that does not work we'd need to chat next week.

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsassoftware%2Fviya4-deployment%2Fissues%2F87%23issuecomment-841317736&data=04%7C01%7Cmarc.leprince%40sas.com%7Cfd9e8db917d8426e696908d916ed815f%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C637566031600097575%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=eQ6KaTf%2FlDxkiKK3OBNgwzo9Jmgj%2F7PgycrgeyTQpfo%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAT7FTQGEUU52DYHFXJMEGWTTNU7BLANCNFSM44HMQUWA&data=04%7C01%7Cmarc.leprince%40sas.com%7Cfd9e8db917d8426e696908d916ed815f%7Cb1c14d5c362545b3a4309552373a0c2f%7C0%7C0%7C637566031600107570%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=B5Bq6p2orrwHi4pHCoKwACjN72RQgU4ieN2QDvvwU38%3D&reserved=0.

[thpang]

@Marc-sas is this still an issue? Can this one be closed? You never reached out to follow-up.

[mleprince018]

Hi Thomas, yes this is. I've been sidetracked between life, vacations and now a big client demo. I'll reach out end of June - maybe? Final demo is June 21st.

[sudipta-sahu]

I am also facing the same error. I tried modifying the main.yaml file with V4_CFG_CR_HOST: "\\"{{ V4_CFG_CR_URL | regex_replace('^https?:\/\/(.)\/?', '\\\\1' }}\"" value however ended up with another error: fatal: [localhost]: FAILED! => {"msg": "The conditional check 'V4_CFG_CR_HOST != \"cr.sas.com\"' failed. The error was: error while evaluating conditional (V4_CFG_CR_HOST != \"cr.sas.com\"): 'V4_CFG_CR_HOST' is undefined\n\nThe error appears to be in '/opt/sas/sas/viya4-deployment/roles/vdm/tasks/main.yaml': line 77, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Include Mirror\n ^ here\n"}

[enderm]

another occurrence has been reported in #133