Normalization of all bash scripts

[Pshepp]

We need to normalize all of our bash scripts. This includes:

• Normalization and inclusion of correct shabangs in each script
• Ensuring all file extensions are the same (so in files with a bash shabang, the file extension for these should be .bash). This can cause some messing with how the git history is tracked, so file extensions are not super high prority but we still need to hit this
• Correct usage of return and exit within our bash commands.
• Utilization of shellformat and shell check. The specific usage is below:
  • shellcheck --enable=require-variable-braces,quote-safe-variables,add-default-case <fileToLintHere>
  • shfmt -i 4 -ci -fn <fileToFormatHere>

Bug Description: A recent bug caused by this was the gems/tests/run_tests.sh script having a #!/bin/sh shabang which caused it to not have the bash source command available to it. Furthermore, inconsistencies increases brain usage which is bad and makes scripts that would grab all specific files etc. to run tools on them more difficult.

Additional context Think of different shells as kinda like different programming langauges. They are not all interchangeable, so we gotta keep that in mind. If we just stick to bash shells that use the baseline commands/progs we should be good. Getting a collection of all our scripts without shabangs would be good for someones first issue, actually changing them and ensuring the shabangs are correct is dependent on the devs skill level

Information regarding shabangs:

• https://stackoverflow.com/questions/16365130/what-is-the-difference-between-usr-bin-env-bash-and-usr-bin-bash
• https://stackoverflow.com/questions/21612980/why-is-usr-bin-env-bash-superior-to-bin-bash
• https://stackoverflow.com/questions/10376206/what-is-the-preferred-bash-shebang

Information Regarding exit vs. return

• Basically, exit exits the script while return returns a code. From my understanding, we should use return in our functions, and ngl we can use exit in our functions if the script is supposed to exit if a function fails but this pattern isnt very common. So let's just go for return on all our functions while using exit to react to a failed functions return value
• https://stackoverflow.com/questions/4419952/difference-between-return-and-exit-in-bash-functions
• https://tldp.org/LDP/abs/html/exit-status.html
• https://www.baeldung.com/linux/return-vs-exit

[gitoliver]

What should everything be

!/bin/bash?

!/usr/bin/env bash?

[Pshepp]

as with most code bits, it depends lolol. These SO posts have a good starting point to understanding, and not just the top comments have decent info:

• https://stackoverflow.com/questions/16365130/what-is-the-difference-between-usr-bin-env-bash-and-usr-bin-bash
• https://stackoverflow.com/questions/21612980/why-is-usr-bin-env-bash-superior-to-bin-bash
• https://stackoverflow.com/questions/10376206/what-is-the-preferred-bash-shebang

I would say just using the following would be fine. Now, as one of the dudes in the 2nd link said on different linoox systems it wont work depending on how people have stuff installed but meh.......

#!/bin/bash

We can also use the following, but it is less secure iirc but idk how much of this being less secure is bs.....

#!usr/bin/env bash

@thecoulter plz weigh in

[Pshepp]

did some edits to the issue bit so i dont have to create a whole other issue. Gonna also be putting this info in glyvemind

[Lachele]

How about for Python and the various environments? It is typical to have several python versions on any given system. If you specify /path/to/this/python you might cause issues.

The use of /bin/bash is problematic for Mac users.

I can probably be convinced otherwise, but I think we should prefer /usr/bin/env unless a certain file is used almost exclusively inside a container and it is a high security risk (and if so, can we make it not be?).

[Pshepp]

yea i was thinking about us properly using conda for our python stuff instead of using a python venv but ngl unsure how useful that would be and how nice it would play with out stack plus wanted to do that in another ticket once we get the bash scripts normalized. That way our conda could force the python version and handle what pip does, but again im not super confident that this will play nice with our current stack (https://stackoverflow.com/questions/73846871/how-to-exchange-data-between-two-different-python-processes-of-different-conda-e is an example of something annoying we would probably have to do with conda). The biggest problem about not normalizing python versions is that some versions of python change how the code itself works. Think back to when we switched from python-3.7 to python-3.9 and some of the lines that messed with strings ended up being invalid when run with python-3.9 thus having a set normalized python version that is what we strive to always use would be nice to just reduce logical overhead that would have to float in devs brains which is something that cannot be understated.

Bash on mac should be easy enough. I am pretty sure that it is already within the OS and if it isnt one can use homebrew aka running brew install bash and im willing to bet u can specifiy a bash version with that.

Using #!/usr/bin/env bash for the shabang doesnt reveal horrifying vulns and after thinking about it it would probably be better for portability sake so i am game with it

[Lachele]

For Mac, there is a built-in bash, but it's an old version of bash. Yao ran into this. The version answering at /bin/bash was too old to run some of our scripts. And, while he could install another bash, he couldn't change the version answering at /bin/bash. So, unless there is a strong reason not to use /usr/bin/env, I think that is he way that is best for us to go.