Fix make sort

[ghost]

The current version in master deletes all the content of the publications-*.bib files on my machine. With the fix it works.

[marcfehling]

That's weird. What's the configuration of your machine? Which OS and shell are you using? Is bibtool called? If yes, does it report any errors?

[ghost]

I am using Win10 with WSL+Ubuntu 22.04.1LTS (this week newly installed). I had a similar problem using the texlive/texlive:latest Dockerimage (as of 22/10/10) and this was the reason for adapting it. I am typing the following commands and after them all publications-*.bib files are empty.

fabian@Laptop:~$ cd git/dealii/publication-list/
fabian@Laptop:~/git/dealii/publication-list$ bibtool -V
BibTool Vers. 2.68 (C) 1996-2019 Gerd Neugebauer

Library path: .:/usr/share/bibtool
Special configuration options: kpathsea
fabian@Laptop:~/git/dealii/publication-list$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
fabian@Laptop:~/git/dealii/publication-list$ make sort

*** BibTool WARNING: File publications-1998.bib not found.

*** BibTool WARNING: File publications-1999.bib not found.

*** BibTool WARNING: File publications-2000.bib not found.

*** BibTool WARNING: File publications-2001.bib not found.

*** BibTool WARNING: File publications-2002.bib not found.

*** BibTool WARNING: File publications-2003.bib not found.

*** BibTool WARNING: File publications-2004.bib not found.

*** BibTool WARNING: File publications-2005.bib not found.

*** BibTool WARNING: File publications-2006.bib not found.

*** BibTool WARNING: File publications-2007.bib not found.

*** BibTool WARNING: File publications-2008.bib not found.

*** BibTool WARNING: File publications-2009.bib not found.

*** BibTool WARNING: File publications-2010.bib not found.

*** BibTool WARNING: File publications-2011.bib not found.

*** BibTool WARNING: File publications-2012.bib not found.

*** BibTool WARNING: File publications-2013.bib not found.

*** BibTool WARNING: File publications-2014.bib not found.

*** BibTool WARNING: File publications-2015.bib not found.

*** BibTool WARNING: File publications-2016.bib not found.

*** BibTool WARNING: File publications-2017.bib not found.

*** BibTool WARNING: File publications-2018.bib not found.

*** BibTool WARNING: File publications-2019.bib not found.

*** BibTool WARNING: File publications-2020.bib not found.

*** BibTool WARNING: File publications-2021.bib not found.

*** BibTool WARNING: File publications-2022.bib not found.

[marcfehling]

fabian@Laptop:~/git/dealii/publication-list$ make sort

*** BibTool WARNING: File publications-1998.bib not found.

To me, this sounds like for f in publications-*.bib ; indeed picks up the right filenames, but for some reason these files either do not exist anymore when passed to bibtool, or bibtool can't find them.

Let's get to the bottom of this.

What output do you get when you add a echo $$f line in either version of the for loop? I get the following:

publications-1998.bib     with   for f in publications-*.bib
./publications-1998.bib   with   for f in $(shell find . -type f -regex ".*\.bib")

What happens when you call bibtool with these filenames directly? Could it also be that there is something wrong with the regex expressions?

[ghost]

For reproducible testing I created the following CI job, which runs on our GitLab Runners. It creates the same error output.

################################################################################
## Check dealii/publication-list with bibtool on ubuntu:jammy
################################################################################
dealii-jammy-format:
  stage: check

  image: ubuntu:jammy

  interruptible: true

  tags:
    - latex

  before_script:
    - apt-get update
    - apt-get install -y bibtool git make ca-certificates --no-install-recommends

  script:
    - git clone https://github.com/dealii/publication-list.git
    - cd publication-list
    - make sort
    - bibtool -r bibtool.rsc -i publications-1998.bib -o publications-1998.bib
    - bibtool -r bibtool.rsc -i ./publications-1999.bib -o ./publications-1999.bib
    - |
      for f in publications-*.bib ; do
        echo $$f
      done
    - echo "$(find . -type f -regex ".*\.bib")"

The following output is generated:

$ git clone https://github.com/dealii/publication-list.git
Cloning into 'publication-list'...
$ cd publication-list
$ make sort
*** BibTool WARNING: File publications-1998.bib not found.
*** BibTool WARNING: File publications-1999.bib not found.
*** BibTool WARNING: File publications-2000.bib not found.
*** BibTool WARNING: File publications-2001.bib not found.
*** BibTool WARNING: File publications-2002.bib not found.
*** BibTool WARNING: File publications-2003.bib not found.
*** BibTool WARNING: File publications-2004.bib not found.
*** BibTool WARNING: File publications-2005.bib not found.
*** BibTool WARNING: File publications-2006.bib not found.
*** BibTool WARNING: File publications-2007.bib not found.
*** BibTool WARNING: File publications-2008.bib not found.
*** BibTool WARNING: File publications-2009.bib not found.
*** BibTool WARNING: File publications-2010.bib not found.
*** BibTool WARNING: File publications-2011.bib not found.
*** BibTool WARNING: File publications-2012.bib not found.
*** BibTool WARNING: File publications-2013.bib not found.
*** BibTool WARNING: File publications-2014.bib not found.
*** BibTool WARNING: File publications-2015.bib not found.
*** BibTool WARNING: File publications-2016.bib not found.
*** BibTool WARNING: File publications-2017.bib not found.
*** BibTool WARNING: File publications-2018.bib not found.
*** BibTool WARNING: File publications-2019.bib not found.
*** BibTool WARNING: File publications-2020.bib not found.
*** BibTool WARNING: File publications-2021.bib not found.
*** BibTool WARNING: File publications-2022.bib not found.
$ bibtool -r bibtool.rsc -i publications-1998.bib -o publications-1998.bib
*** BibTool WARNING: File publications-1998.bib not found.
$ bibtool -r bibtool.rsc -i ./publications-1999.bib -o ./publications-1999.bib
$ for f in publications-*.bib ; do # collapsed multi-line command
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
1f
$ echo "$(find . -type f -regex ".*\.bib")"
./publications-2000.bib
./publications-2021.bib
./publications-2016.bib
./publications-2014.bib
./publications-2002.bib
./publications-2018.bib
./publications-2009.bib
./publications-2019.bib
./publications-2013.bib
./publications-2012.bib
./publications-2015.bib
./publications-2010.bib
./publications-2006.bib
./publications-1998.bib
./publications-2008.bib
./publications-2007.bib
./publications-2004.bib
./publications-2020.bib
./publications-2022.bib
./publications-2001.bib
./publications-1999.bib
./publications-2017.bib
./publications-2005.bib
./publications-2011.bib
./publications-2003.bib

I do not understand what is the output 1f.

Comparing:

publications-1998.bib with for f in publications-\*.bib ./publications-1998.bib with for f in $(shell find . -type f -regex ".*\.bib")

Gives the same in the Dockerimage.

bibtool works with filenames as ./publications-1998.bib but seems not to work without ./ prefix.

[marcfehling]

for f in publications-*.bib ; do
  echo $$f
done

Only in the Makefile you are going to use $$f, in a general shell you need $f.

---

bibtool -r bibtool.rsc -i publications-1998.bib -o publications-1998.bib
*** BibTool WARNING: File publications-1998.bib not found.

This is weird. Just out of curiosity, can you call pwd and ls in your script as well and post the output here? Maybe somehow the working directory is screwed up?

---

Which shell are you using?

[marcfehling]

Looks like a similar problem has been posted here as well: https://github.com/ge-ne/bibtool/issues/87/

[ghost]

Looks like a similar problem has been posted here as well: ge-ne/bibtool#87

In my case it stands: Library path: .:/usr/share/bibtool. So the ./ may be mandatory?

[ghost]

for f in publications-*.bib ; do
  echo $$f
done

Only in the Makefile you are going to use $$f, in a general shell you need $f.

bibtool -r bibtool.rsc -i publications-1998.bib -o publications-1998.bib
*** BibTool WARNING: File publications-1998.bib not found.

This is weird. Just out of curiosity, can you call pwd and ls in your script as well and post the output here? Maybe somehow the working directory is screwed up?

Which shell are you using?

The output of pwd and ls and the fixed echo $f command is:

$ pwd
/builds/references/publication-list
$ ls
Jenkinsfile
Makefile
README.md
bibtool.rsc
jabref-template
offline
publications-1998.bib
publications-1999.bib
publications-2000.bib
publications-2001.bib
publications-2002.bib
publications-2003.bib
publications-2004.bib
publications-2005.bib
publications-2006.bib
publications-2007.bib
publications-2008.bib
publications-2009.bib
publications-2010.bib
publications-2011.bib
publications-2012.bib
publications-2013.bib
publications-2014.bib
publications-2015.bib
publications-2016.bib
publications-2017.bib
publications-2018.bib
publications-2019.bib
publications-2020.bib
publications-2021.bib
publications-2022.bib
publications.include
$ for f in publications-*.bib ; do # collapsed multi-line command
publications-1998.bib
publications-1999.bib
publications-2000.bib
publications-2001.bib
publications-2002.bib
publications-2003.bib
publications-2004.bib
publications-2005.bib
publications-2006.bib
publications-2007.bib
publications-2008.bib
publications-2009.bib
publications-2010.bib
publications-2011.bib
publications-2012.bib
publications-2013.bib
publications-2014.bib
publications-2015.bib
publications-2016.bib
publications-2017.bib
publications-2018.bib
publications-2019.bib
publications-2020.bib
publications-2021.bib
publications-2022.bib
$ echo "$(find . -type f -regex ".*\.bib")"
./publications-2000.bib
./publications-2021.bib
./publications-2016.bib
./publications-2014.bib
./publications-2002.bib
./publications-2018.bib
./publications-2009.bib
./publications-2019.bib
./publications-2013.bib
./publications-2012.bib
./publications-2015.bib
./publications-2010.bib
./publications-2006.bib
./publications-1998.bib
./publications-2008.bib
./publications-2007.bib
./publications-2004.bib
./publications-2020.bib
./publications-2022.bib
./publications-2001.bib
./publications-1999.bib
./publications-2017.bib
./publications-2005.bib
./publications-2011.bib
./publications-2003.bib

I am using a Dockerimage and suppose that it is a typical bash shell.

[marcfehling]

That doesn't sound right. Can you report to bibtool that you encountered this behavior? Provide them a minimal example that works when calling bibtool ./publication.bib and fails with bibtool publication.bib, and specify your system configuration with as much detail as possible (as none of us can recreate that behavior). I think you can just continue the already opened issue in the bibtool repository.

Maybe they can get it fixed. In the meantime, we should consider merging your patch. @bangerth -- What do you think?

[marcfehling]

In my case it stands: Library path: .:/usr/share/bibtool. So the ./ may be mandatory?

For me, bibtool -h reports Library path: .:/usr/lib/BibTool. So it is similar to yours, but mine works without the ./

[bangerth]

/rebuild

[ghost]

I'm ok with this -- thank you. If you squash/fixup the two commits, I will merge.

Now it is rebased and squashed. Thank you!

[ghost]

That doesn't sound right. Can you report to bibtool that you encountered this behavior? Provide them a minimal example that works when calling bibtool ./publication.bib and fails with bibtool publication.bib, and specify your system configuration with as much detail as possible (as none of us can recreate that behavior). I think you can just continue the already opened issue in the bibtool repository.

Maybe they can get it fixed. In the meantime, we should consider merging your patch. @bangerth -- What do you think?

I created a minimal example, which produces this error and added a comment to the issue there.