"getbib" script fails to get the doi addresses from many files. Especially when the doi address is given with a URL instead of directly given.
We can use only a single line of "sed" command to get the desired output on almost any kind of given doi addresses while using an even more minimal and clean command.
Examples & Very Detailed Explanations Below
Here are the examples before and after:
argument1="doi.org/10.1038/s41594-023-00968-y."
argument2="doi.org/10.1038/s41594-023-00968-y"
argument3="https://doi.org/10.1038/s41594-023-00968-y."
argument4="https://doi.org/10.1038/s41594-023-00968-y"
argument5="doi: 10.1038/s41594-023-00968-y"
argument6="doi: 10.1038/s41594-023-00968-y."
argument7="doi:10.1038/s41594-023-00968-y."
argument8="doi.org/10.1038/s41594-023-00968-5."
argument9="doi.org/10.1038/s41594-023-00968-5"
argument10="https://doi.org/10.1038/s41594-023-00968-5."
argument11="https://doi.org/10.1038/s41594-023-00968-5"
argument12="doi: 10.1038/s41594-023-00968-8"
argument13="doi: 10.1038/s41594-023-00968-3"
argument14="doi: 10.1038/s41594-023-00968-3."
argument15="doi:10.1038/s41594-023-00968-3."
argument16=".doi:10.1038/s41594-023-00968-3."
argument17="adoi:10.1038/s41594-023-00968-3."
argument18="a: doi:10.1038/s41594-023-00968-3."
argument19="a doi:10.1038/s41594-023-00968-3."
argument20="doi: a0.1038/s41594-023-00968-3"
argument21="doi: b01038/s41594-023-009.68-3."
argument22="doi:c01038/s41594-023-00968-3."
argument23=".doi:d01038/s4.1594-023-00968-3."
argument24="adoi:e01.038/s41594-023-00968-3."
argument25="a: doi:f010.38/s41594-023-00968-3."
argument26="a doi:g0103.8/s41594-023-00968-3."
The Output from the Old Command(Only Gives 18 outputs out of 26 examples and Every Output is incorrect from my examples):
10.1038/s41594-023-00968-y
10.1038/s41594-023-00968-y
10.1038/s41594-023-00968-3
10.1038/s41594-023-00968-8
10.1038/s41594-023-00968-3
10.1038/s41594-023-00968-3
10.1038/s41594-023-00968-3
10.1038/s41594-023-00968-3
10.1038/s41594-023-00968-3
10.1038/s41594-023-00968-3
10.1038/s41594-023-00968-3
c01038/s41594-023-00968-33
b01038/s41594-023-009.68-3
c01038/s41594-023-00968-33
d01038/s4.1594-023-00968-3
e01.038/s41594-023-00968-3
f010.38/s41594-023-00968-3
g0103.8/s41594-023-00968-3
As it can be seen, they fail to start with "doi:" and some of them give 0 output.
The Example Output After the Change to the New Command
Very Detailed Explanation (I realized that escaped backslashes do not appear. There is a backslash if you see nothing.)
(For people who wonder about it, or try to learn. It could take a tremendous amount of time to learn all of it without explanation, so it would be better to explain):
sed The sed command is a stream editor that can be used to perform basic text transformations on an input file or from a pipeline. You can see Luke uses it a lot in his videos. It can also modify files' content if you want for other purposes. That function is used a lot for bootstrapping scripts for changing config files automatically if necessary.
-n This option tells sed not to print lines by default. We'll only print lines when we specify the p command in the script.
-E This option enables the use of extended regular expressions, which allows for more readable and flexible regex patterns.
's/ This starts the sed script and defines the s command (substitute). It is used to find a regex pattern in the input and replace it with a specified string.
**.*** This regex pattern matches any character (except a newline) zero or more times. In this case, it matches all characters before "doi" or "DOI".
( This paranthesis opens a capturing group, which allows us to refer back to the matched text later in the script.
(DOI|doi) This regex pattern matches either "DOI" or "doi". The | symbol is used as an OR operator in regular expressions.
( This next paranthesis opens another capturing group.
(.(org))? This regex pattern matches an optional ".org". The . is an escaped period, and (org) matches the string "org". The ? following the group makes it optional. Escaping is needed for most of non-alphanumeric characters. You can test and practice them on vim, trying to use the "substitute" function to change some text.
\/? This regex pattern matches an optional "/", with the ? making it optional. The prior backslash is for escaping. Again, some characters need to be escaped to be able to used in commands. Escaped means they have \ before them. Spaces may be the most escaped characters.
| This symbol, later, also acts as an OR operator, indicating that the pattern before or after it can be matched.
:? This regex pattern matches an optional colon (":") followed by zero or more spaces. The ? makes the colon optional, and matches zero or more spaces.
) This closes the capturing group started earlier.
) This closes the outer capturing group.
([^: ]+[^ .]) This regex pattern matches any character except colons and spaces one or more times ([^: ]+) Plus symbol here shows one or more times. If it is a star then it means zero or more times. It is then followed by a single alphanumeric character ([^ .]) Single because there are no plus or star symbol next to it. This part as a whole ensures that the last character of the matched text is alphanumeric.
**.*** This regex pattern matches any character (except a newline) zero or more times. In this case, it matches all remaining characters in the input line.
/ This delimiter separates the regex pattern from the replacement string in the s command. s command needs a separator that is a forward slash.
doi:\6 This is the replacement string. The text "doi:" is followed by the 6th captured group from the regex pattern, which contains the characters after "doi" or "DOI" and the colon, "/", or space(s).
/p This delimiter separates the replacement string from the p command, which tells sed to print the modified line if a substitution has been made. The substitution mentioned here is the change of ".org/" to ":". This helps turning URLs into doi addresses.
; This separates different commands within the sed script.
T This command branches to the end of the script if no substitution was made since the last input line was read or conditional branch was taken. In this case, it ensures that the q command is only executed if a matching line has been found and a substitution was made. This is one of the most important parts to get the doi address from the urls such as "https://doi.org/10.1038/s41594-023-00968-5". Because we don't always have URLs for doi addresses. In this way, this function only works when we work with URLs. So in this case it helps changing .org/ with : This makes the part of the doi address as this: "doi:" rather than this: "doi.org/".
q This command tells sed to quit processing after the first match, ensuring that only the first matching line in the file is processed. Otherwise, we would get all doi addresses in a scientific study because there are lots of doi addresses in them.
' This closes the other '
TL;DR:
Basically this whole command ensures that the output we get starts with "doi:", then it can have every type of character in it except spaces and ".org/" , then it will end with an alphanumeric character [A-Z, a-z or 0-9]. That ensures removing the trailing dots from some doi addresses that have them.
"getbib" script fails to get the doi addresses from many files. Especially when the doi address is given with a URL instead of directly given.
We can use only a single line of "sed" command to get the desired output on almost any kind of given doi addresses while using an even more minimal and clean command.
Examples & Very Detailed Explanations Below
Here are the examples before and after: argument1="doi.org/10.1038/s41594-023-00968-y." argument2="doi.org/10.1038/s41594-023-00968-y" argument3="https://doi.org/10.1038/s41594-023-00968-y." argument4="https://doi.org/10.1038/s41594-023-00968-y" argument5="doi: 10.1038/s41594-023-00968-y" argument6="doi: 10.1038/s41594-023-00968-y." argument7="doi:10.1038/s41594-023-00968-y." argument8="doi.org/10.1038/s41594-023-00968-5." argument9="doi.org/10.1038/s41594-023-00968-5" argument10="https://doi.org/10.1038/s41594-023-00968-5." argument11="https://doi.org/10.1038/s41594-023-00968-5" argument12="doi: 10.1038/s41594-023-00968-8" argument13="doi: 10.1038/s41594-023-00968-3" argument14="doi: 10.1038/s41594-023-00968-3." argument15="doi:10.1038/s41594-023-00968-3." argument16=".doi:10.1038/s41594-023-00968-3." argument17="adoi:10.1038/s41594-023-00968-3." argument18="a: doi:10.1038/s41594-023-00968-3." argument19="a doi:10.1038/s41594-023-00968-3." argument20="doi: a0.1038/s41594-023-00968-3" argument21="doi: b01038/s41594-023-009.68-3." argument22="doi:c01038/s41594-023-00968-3." argument23=".doi:d01038/s4.1594-023-00968-3." argument24="adoi:e01.038/s41594-023-00968-3." argument25="a: doi:f010.38/s41594-023-00968-3." argument26="a doi:g0103.8/s41594-023-00968-3."
The Output from the Old Command (Only Gives 18 outputs out of 26 examples and Every Output is incorrect from my examples): 10.1038/s41594-023-00968-y 10.1038/s41594-023-00968-y 10.1038/s41594-023-00968-3 10.1038/s41594-023-00968-8 10.1038/s41594-023-00968-3 10.1038/s41594-023-00968-3 10.1038/s41594-023-00968-3 10.1038/s41594-023-00968-3 10.1038/s41594-023-00968-3 10.1038/s41594-023-00968-3 10.1038/s41594-023-00968-3 c01038/s41594-023-00968-33 b01038/s41594-023-009.68-3 c01038/s41594-023-00968-33 d01038/s4.1594-023-00968-3 e01.038/s41594-023-00968-3 f010.38/s41594-023-00968-3 g0103.8/s41594-023-00968-3
As it can be seen, they fail to start with "doi:" and some of them give 0 output.
The Example Output After the Change to the New Command
(Every Output is correct.):
doi:10.1038/s41594-023-00968-y doi:10.1038/s41594-023-00968-y doi:10.1038/s41594-023-00968-y doi:10.1038/s41594-023-00968-y doi:10.1038/s41594-023-00968-y doi:10.1038/s41594-023-00968-y doi:10.1038/s41594-023-00968-y doi:10.1038/s41594-023-00968-5 doi:10.1038/s41594-023-00968-5 doi:10.1038/s41594-023-00968-8 doi:10.1038/s41594-023-00968-5 doi:10.1038/s41594-023-00968-8 doi:10.1038/s41594-023-00968-3 doi:10.1038/s41594-023-00968-3 doi:10.1038/s41594-023-00968-3 doi:10.1038/s41594-023-00968-3 doi:10.1038/s41594-023-00968-3 doi:10.1038/s41594-023-00968-3 doi:10.1038/s41594-023-00968-3 doi:a0.1038/s41594-023-00968-3 doi:d01038/s4.1594-023-00968-3 doi:c01038/s41594-023-00968-3 doi:d01038/s4.1594-023-00968-3 doi:e01.038/s41594-023-00968-3 doi:f010.38/s41594-023-00968-3 doi:g0103.8/s41594-023-00968-3
Very Detailed Explanation (I realized that escaped backslashes do not appear. There is a backslash if you see nothing.)
(For people who wonder about it, or try to learn. It could take a tremendous amount of time to learn all of it without explanation, so it would be better to explain):
sed The sed command is a stream editor that can be used to perform basic text transformations on an input file or from a pipeline. You can see Luke uses it a lot in his videos. It can also modify files' content if you want for other purposes. That function is used a lot for bootstrapping scripts for changing config files automatically if necessary.
-n This option tells sed not to print lines by default. We'll only print lines when we specify the p command in the script.
-E This option enables the use of extended regular expressions, which allows for more readable and flexible regex patterns.
's/ This starts the sed script and defines the s command (substitute). It is used to find a regex pattern in the input and replace it with a specified string.
**.*** This regex pattern matches any character (except a newline) zero or more times. In this case, it matches all characters before "doi" or "DOI".
( This paranthesis opens a capturing group, which allows us to refer back to the matched text later in the script.
(DOI|doi) This regex pattern matches either "DOI" or "doi". The | symbol is used as an OR operator in regular expressions.
( This next paranthesis opens another capturing group.
(.(org))? This regex pattern matches an optional ".org". The . is an escaped period, and (org) matches the string "org". The ? following the group makes it optional. Escaping is needed for most of non-alphanumeric characters. You can test and practice them on vim, trying to use the "substitute" function to change some text.
\/? This regex pattern matches an optional "/", with the ? making it optional. The prior backslash is for escaping. Again, some characters need to be escaped to be able to used in commands. Escaped means they have \ before them. Spaces may be the most escaped characters.
| This symbol, later, also acts as an OR operator, indicating that the pattern before or after it can be matched.
:? This regex pattern matches an optional colon (":") followed by zero or more spaces. The ? makes the colon optional, and matches zero or more spaces.
) This closes the capturing group started earlier.
) This closes the outer capturing group.
([^: ]+[^ .]) This regex pattern matches any character except colons and spaces one or more times ([^: ]+) Plus symbol here shows one or more times. If it is a star then it means zero or more times. It is then followed by a single alphanumeric character ([^ .]) Single because there are no plus or star symbol next to it. This part as a whole ensures that the last character of the matched text is alphanumeric.
**.*** This regex pattern matches any character (except a newline) zero or more times. In this case, it matches all remaining characters in the input line.
/ This delimiter separates the regex pattern from the replacement string in the s command. s command needs a separator that is a forward slash.
doi:\6 This is the replacement string. The text "doi:" is followed by the 6th captured group from the regex pattern, which contains the characters after "doi" or "DOI" and the colon, "/", or space(s).
/p This delimiter separates the replacement string from the p command, which tells sed to print the modified line if a substitution has been made. The substitution mentioned here is the change of ".org/" to ":". This helps turning URLs into doi addresses.
; This separates different commands within the sed script.
T This command branches to the end of the script if no substitution was made since the last input line was read or conditional branch was taken. In this case, it ensures that the q command is only executed if a matching line has been found and a substitution was made. This is one of the most important parts to get the doi address from the urls such as "https://doi.org/10.1038/s41594-023-00968-5". Because we don't always have URLs for doi addresses. In this way, this function only works when we work with URLs. So in this case it helps changing .org/ with : This makes the part of the doi address as this: "doi:" rather than this: "doi.org/".
q This command tells sed to quit processing after the first match, ensuring that only the first matching line in the file is processed. Otherwise, we would get all doi addresses in a scientific study because there are lots of doi addresses in them.
' This closes the other '
TL;DR:
Basically this whole command ensures that the output we get starts with "doi:", then it can have every type of character in it except spaces and ".org/" , then it will end with an alphanumeric character [A-Z, a-z or 0-9]. That ensures removing the trailing dots from some doi addresses that have them.