bib file not found even though known to kpse

[AndrewSwann]

Running biber 2.18 on a mac, installed via macports texlive distribution, bib files that can be found by kpsewhich in the shell are not longer found by biber. This was previously working for me a few days ago.

Running with --trace, I get

[0] Config.pm:306> INFO - This is Biber 2.18
[0] Config.pm:309> INFO - Logfile is 'a.blg'
[80] biber-darwin:340> INFO - === Wed Jun 29, 2022, 08:39:55
[97] Biber.pm:418> INFO - Reading 'a.bcf'
[196] Biber.pm:978> INFO - Found 1 citekeys in bib section 0
[196] Biber.pm:981> DEBUG - The citekeys for section 0 are: test
[323] biber-darwin:426> TRACE - 
....
[335] Biber.pm:4401> INFO - Processing section 0
[335] Biber.pm:4515> DEBUG - Looking for directly cited keys: test
[347] Biber.pm:4592> INFO - Looking for bibtex file 'x.bib' for section 0
[347] Utils.pm:410> ERROR - Cannot find 'x.bib'!
[347] Biber.pm:135> INFO - ERRORS: 1

Looking at the source of Utils.pm suggests that kpsewhich was never tried, otherwise the report should have included messages such as Looking for file '$sourcepath' via kpsewhich etc. I have tried removing the biber cache and running again, but this does not change the result.

[plk]

Hmm, that's the second report of this I've had but currently I can't reproduce it. You are right that it seems that it can't find kpsewhich in the PATH for some reason. Is this on x86 or ARM?

[AndrewSwann]

It is x86.

[plk]

Just to make sure we're comparing the same thing, can you download onto your desktop the universal binary from here:

https://sourceforge.net/projects/biblatex-biber/files/biblatex-biber/current/binaries/MacOS/biber-darwin_universal.tar.gz/download

(you may well have to clear the security complaint as this binary isn't signed like the TL one - try to open it once, close the complaint window, go into System Preferences->Security & Privacy and you'll see biber mentioned with a "Allow Anyway" button. Click that and on the next attempt to run it, just click "Open" in the popup window and after that it will run normally)

and then run:

~/Desktop/biber --debug --tool biblatex-examples.bib

and then look in the biblatex-examples.bib.blg

to see what's happening?

I just did this on an x86 laptop and it's fine for me.

Also check to make sure kpsewhich is in the PATH of the shell running biber?

[AndrewSwann]

Many thanks, that downloaded version works. The version on my system with the corresponding invocation fails. kpsewhich runs fine from the same shell. tlmgr info biber gives

package:     biber
category:    Package
shortdesc:   A BibTeX replacement for users of BibLaTeX
longdesc:    Biber is a BibTeX replacement for users of BibLaTeX. Biber supports full UTF-8, can (re)-encode input and output, supports highly configurable sorting, dynamic bibliography sets and many other features. The CTAN distribution offers a compressed tar archive of the sources, etc., together with "binary" distributions for a variety of platforms. Note: on SourceForge biber is formally named "biblatex-biber", to distinguish it from an earlier (now apparently moribund) project called "biber".
installed:   Yes
revision:    63708
sizes:       src: 1193k, doc: 269k, bin: 98113k
relocatable: No

and tlmgr restore provides

tlmgr restore --dry-run biber
tlmgr restore: dry run, no changes will be made
Available backups for biber: 61867 (2022-06-26 11:52)

So the texlive version has been bumped recently. Should I try to restore this earlier version?

[plk]

Would be interesting to see what difference that makes - I have no idea what is going on here currently - I just released the MacOS stuff as usual and it was pulled into TL.

[AndrewSwann]

OK, I have now tried that. Unfortunately, restoring the texlive version 61867 doesn't help. Should I/we be asking the texlive maintainers instead? Incidentally reupdating texlive now gives revision number 63752, but it still has the same problem.

[plk]

I think the move to 6308 was just a documentation change as I had to request that yesterday. I just tried the same thing using the TL binary from macTeX TL 2021 and 2022 on both ARM and x86 and I still can't get it to fail. Tried version 61867 and 63708 and both work for me on ARM and x86_64. Very odd.

[AndrewSwann]

Is there live dependence on perl modules or is everything included in the binary? I notice some parts of my perl installation have updated recently, but I haven't used perl for a long time, so am not sure what to check. The can_run command is from IPC:Cmd. But that works fine

#!/usr/bin/env perl

use IPC::Cmd qw( can_run );
print(can_run('kpsewhich'));

gives

/Library/TeX/texbin/kpsewhich%  

as output

[plk]

There are no live dependencies - it's all internal to the cached perl tree "inside" biber. I was thinking it might be to do with what they do to the binaries to sign them in TL as that one you downloaded was the exact binary they pulled into CTAN and TL. They then have to sign them for MacOS. However, that still doesn't explain why I can run all of this without problems with the TL binaries ...

[plk]

Can you let me know which version of MacOS this is on?

[herbs]

I'm on macOS 12.4, Monterey. I'm also using a fully updated TeX Live 2022 originally installed via MacTeX.

By the way, in an earlier email you noted that there were 2 versions of kpsewhich on your system. One was in /opt via MacPorts. The other seemed to be in TeX Live 2021 rather than 2022! I have only one active version of kpsewhich on my PATH.

[plk]

That situation with macports kpsewhich was a red herring I think. It's currently frustrating that I can't reproduce this. I also have TL 2022 via MacTeX on an x86 laptop but it's not 12.4 as it's an older laptop that can't take 12. I have 12.4 on ARM but that doesn't seem to be an issue. Let's see what @AndrewSwann has.

[AndrewSwann]

I am also on macOS 12.4 Monterey. biber was working fine a week or so ago and I have been running Monterey for some time. I have only one version of kpsewhich on my system: /Library/TeX/texbin/kpsewhich

[plk]

Can you both try deleting the biber cache and trying this again? To do this, run

biber --cache

and just delete the entire directory this returns. Then, be sure to run biber once until it returns so it can unpack the cache again.

[herbs]

On Jun 30, 2022, at 6:21 AM, plk @.***> wrote:

Can you both try deleting the biber cache and trying this again? To do this, run

biber --cache

and just delete the entire directory this returns. Then, be sure to run biber once until it returns so it can unpack the cache again.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

Howdy,

Sigh... I've done that before (I have a TeXShop Macro that removes the cache) but no luck. It still fails to find the bib file unless the file is in the same folder as the tex file. The only real difference is the longer wait while biber re-creates the cache.

Good Luck,

Herb Schulz @.***

[AndrewSwann]

I removed that entire parent directory to the cache and ran biber again, but unfortunately that doesn't solve the problem.

[plk]

Can you try the following and see what happens?

cp /usr/local/texlive/2022/bin/universal-darwin/biber ~/Desktop/biber
codesign --remove-signature ~/Desktop/biber
~/Desktop/biber --tool biblatex-examples.bib

[plk]

Can you also get this small binary:

https://www.dropbox.com/s/kjocvk15eb0sddi/test?dl=0

and run it on your machines with the problem? It just runs the perl code below to see what environment PAR is seeing:

use IPC::Cmd (can_run);
use v5.26;

say 'PATH=' . $ENV{'PATH'};
say can_run "kpsewhich";

[AndrewSwann]

Moving biber to Desktop as you described, and running worked fine. I have sent you the result of the test script by email.

[plk]

Hmm, did the copy to the Desktop work or was it the codesign signature removal that did the trick?

[AndrewSwann]

Good point. Just copying to the Desktop without removing the signature works.

[plk]

That is really odd - let's see what happens for @herbs.

[plk]

Just out of interest - what happens if you overwrite the TL binary with the one copied to the Desktop?

[herbs]

On Jun 30, 2022, at 8:09 AM, plk @.***> wrote:

Can you try the following and see what happens?

cp /usr/local/texlive/2022/bin/universal-darwin/biber ~/Desktop/biber codesign --remove-signature ~/Desktop/biber ~/Desktop/biber --tool biblatex-examples.bib

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

Howdy,

I had to approve the running of an unsigned app in the `Security & Privacy' tab of System Preferences. Even after approving it zsh killed the run.

Good Luck,

Herb Schulz @.***

[herbs]

On Jun 30, 2022, at 8:17 AM, plk @.***> wrote:

Can you also get this small binary:

https://www.dropbox.com/s/kjocvk15eb0sddi/test?dl=0

and run it on your machines with the problem? It just runs the perl code below to see what environment PAR is seeing:

use IPC::Cmd (can_run); use v5.26;

say 'PATH=' . $ENV{'PATH'}; say can_run "kpsewhich";

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

Howdy,

After approving the running of an unsigned downloaded app as before I got the expected output:

% ./test PATH=/Users/herbs/opt/anaconda3/bin:/Users/herbs/opt/anaconda2/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Applications/Little Snitch.app/Contents/Components:/opt/X11/bin:/Library/Apple/usr/bin /Library/TeX/texbin/kpsewhich

so it's finding kpsewhich correctly.

Good Luck,

Herb Schulz @.***

[plk]

and zsh kills it every time now? Sometimes there is a second popup to allow unsigned after the System Preferences auth.

[plk]

One more test then - what about copying the test binary temporarily to /usr/local/texlive/2022/bin/universal-darwin/? I have a feeling that something horrible is happening with MacOS SIP here.

[AndrewSwann]

Copying it from Desktop back to the texlive location gives the old error

[AndrewSwann]

No, copying to universal-darwin directory does not help.

[plk]

What output does the "test" binary give when it's in /usr/local/texlive/2022/bin/universal-darwin/? Same as when it is in Desktop?

[herbs]

On Jun 30, 2022, at 8:55 AM, plk @.***> wrote:

What output does the "test" binary give when it's in /usr/local/texlive/2022/bin/universal-darwin/? Same as when it is in Desktop?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

Howdy,

I get the same output if I run it from HOME as mytest'. Unfortunately zsh has an internal command calledtest' :-( so I had to give it a different name.

I know that under Monterey (macOS 12) app sources are supposed to be uploaded and check/approved. This is done automatically in the present Xcode. You might want to get in touch with Dick Koch since he has experience with this process. I believe that is what is giving rise to the

dialog box and what requires approval in System Preferences as noted above.

Good Luck,

Herb Schulz @.***

[plk]

I'll get Mr Koch in here if I can - we have already discussed the whole signing/notarizing thing before and I suspect it has to do with this or the SIP enhancements in MacOS 12 on Intel. It is odd however that copying the binary out of the TL tree only works for one of you however. I assume you both have SIP enabled - it would be unless you purposely disabled it - you can check with:

csrutil status

[AndrewSwann]

Yes, SIP is enabled for me.

[plk]

So, there is a workaround for now by just having the binary outside of TL but before TL in your PATH. This looks like a SIP issue. Wish I could reproduce but it looks like something that >10.15.7 introduced. Have invited Richard Koch here for his opinion.

[herbs]

On Jun 30, 2022, at 10:07 AM, plk @.***> wrote:

I'll get Mr Koch in here if I can - we have already discussed the whole signing/notarizing thing before and I suspect it has to do with this or the SIP enhancements in MacOS 12 on Intel. It is odd however that copying the binary out of the TL tree only works for one of you however. I assume you both have SIP enabled - it would be unless you purposely disabled it - you can check with:

csrutil status

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

Howdy,

SIP is enabled here too.

It works on the Desktop after allowing it to run in the Security & Privacy' tab ofSystem Preferences'. Then I first get that dialog box and if I allow it to run there the output is as expected:

% ~/Desktop/biber --tool biblatex-examples.bib INFO - This is Biber 2.18 running in TOOL mode INFO - Logfile is 'biblatex-examples.bib.blg' INFO - Looking for bibtex file 'biblatex-examples.bib' INFO - LaTeX decoding ... INFO - Found BibTeX data source '/usr/local/texlive/2022/texmf-dist/bibtex/bib/biblatex/biblatex/biblatex-examples.bib' INFO - Overriding locale 'en_US' defaults 'variable = shifted' with 'variable = non-ignorable' INFO - Overriding locale 'en_US' defaults 'normalization = NFD' with 'normalization = prenormalized' INFO - Sorting list 'tool/global//global/global' of type 'entry' with template 'tool' and locale 'en_US' INFO - No sort tailoring available for locale 'en_US' INFO - Writing 'biblatex-examples_bibertool.bib' with encoding 'UTF-8' INFO - Output to biblatex-examples_bibertool.bib

Good Luck,

Herb Schulz @.***

[plk]

That's actually good as this is consistent for both of you now. This must be a SIP thing as it makes no sense in normal UNIX land.

[leonmoonen]

Hi,

I have the same issue after updating my texlive a week ago and found this thread while investigating a solution. I'm also on macOS Monterey 12.4, x86, SIP enabled. I have downloaded the binary from Sourceforge and put it in my personal bin folder (which is before texlive in my path). However:

• when I run that biber via the PATH as biber --tool biblatex-examples.bib, it cannot find biblatex-examples.bib
• when I run that biber via an explicit path as ~/bin/biber --tool biblatex-examples.bib, it can find biblatex-examples.bib
• when I run the TexLive installed biber via an explicit path as /Library/TeX/texbin/biber --tool biblatex-examples.bib, it can also find biblatex-examples.bib, and all works as expected

This makes me think it's not a code signing problem of biber but maybe something with how it finds kpsewhich internally? Your test binary was no longer available so I could not test that, but running the code as a perl script gives the expected output.

Also: as Andrew pointed out in the first message, this problem occurs only when the bibtex file is not in the current directory but needs to be found via kpsewhich. If the file is symlinked in the local directory, biber works as expected (this is an easy workaround).

➤ cd ~/tmp
➤ which biber
/Library/TeX/texbin/biber

➤ biber --tool biblatex-examples.bib
INFO - This is Biber 2.18 running in TOOL mode
INFO - Logfile is 'biblatex-examples.bib.blg'
INFO - Looking for bibtex file 'biblatex-examples.bib'
ERROR - Cannot find 'biblatex-examples.bib'!
INFO - ERRORS: 1

➤ cd ~/bin
➤ tar xzf ~/Downloads/biber-darwin_universal.tar.gz
➤ cd ~/tmp
➤ which biber
/Users/leon/bin/biber

➤ biber --tool biblatex-examples.bib
INFO - This is Biber 2.18 running in TOOL mode
INFO - Logfile is 'biblatex-examples.bib.blg'
INFO - Looking for bibtex file 'biblatex-examples.bib'
ERROR - Cannot find 'biblatex-examples.bib'!
INFO - ERRORS: 1

➤ ~/bin/biber --tool biblatex-examples.bib
INFO - This is Biber 2.18 running in TOOL mode
INFO - Logfile is 'biblatex-examples.bib.blg'
INFO - Looking for bibtex file 'biblatex-examples.bib'
INFO - LaTeX decoding ...
INFO - Found BibTeX data source '/usr/local/texlive/2022/texmf-dist/bibtex/bib/biblatex/biblatex/biblatex-examples.bib'
INFO - Overriding locale 'en_US' defaults 'variable = shifted' with 'variable = non-ignorable'
INFO - Overriding locale 'en_US' defaults 'normalization = NFD' with 'normalization = prenormalized'
INFO - Sorting list 'tool/global//global/global' of type 'entry' with template 'tool' and locale 'en_US'
INFO - No sort tailoring available for locale 'en_US'
INFO - Writing 'biblatex-examples_bibertool.bib' with encoding 'UTF-8'
INFO - Output to biblatex-examples_bibertool.bib

➤ /Library/TeX/texbin/biber --tool biblatex-examples.bib
INFO - This is Biber 2.18 running in TOOL mode
INFO - Logfile is 'biblatex-examples.bib.blg'
INFO - Looking for bibtex file 'biblatex-examples.bib'
INFO - LaTeX decoding ...
INFO - Found BibTeX data source '/usr/local/texlive/2022/texmf-dist/bibtex/bib/biblatex/biblatex/biblatex-examples.bib'
INFO - Overriding locale 'en_US' defaults 'normalization = NFD' with 'normalization = prenormalized'
INFO - Overriding locale 'en_US' defaults 'variable = shifted' with 'variable = non-ignorable'
INFO - Sorting list 'tool/global//global/global' of type 'entry' with template 'tool' and locale 'en_US'
INFO - No sort tailoring available for locale 'en_US'
INFO - Writing 'biblatex-examples_bibertool.bib' with encoding 'UTF-8'
INFO - Output to biblatex-examples_bibertool.bib

➤ which kpsewhich
/Library/TeX/texbin/kpsewhich

➤ kpsewhich biblatex-examples.bib
/usr/local/texlive/2022/texmf-dist/bibtex/bib/biblatex/biblatex/biblatex-examples.bib

cheers, Leon

[plk]

Hmm @AndrewSwann, @herbs - can you see if using an absolute path works for you? The problem remains that I can't reproduce this on 10.15.7 Intel.

[plk]

Test binary is back:

https://www.dropbox.com/s/nn2ypny2sh0ivf5/test?dl=0

Can you try PATH and absolute calls of this when it's in /Library/TeX/texbin/

[leonmoonen]

that test binary only has the arm code, I would need the x86:

➤ file ~/Downloads/test
/Users/leon/Downloads/test: Mach-O 64-bit executable arm64

[plk]

Apologies: https://www.dropbox.com/s/zu4ka1hpsjdp8wy/test?dl=0

[herbs]

On Jul 3, 2022, at 10:29 AM, plk @.***> wrote:

Hmm @AndrewSwann, @herbs - can you see if using an absolute path works for you? The problem remains that I can't reproduce this on 10.15.7 Intel.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

Howdy,

Sigh... after today's update to latex and l3 stuff biblatex is temporarily broken. :-(

Supposedly a fixed biblatex (temporary?) is on the way.

Good Luck,

Herb Schulz @.***

[leonmoonen]

gotcha!

➤ biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin

➤ /Library/TeX/texbin/biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin:/usr/local/Caskroom/miniconda/base/condabin:/Users/leon/bin:/usr/local/bin:/usr/local/sbin:/usr/local/opt/sqlite/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
/Library/TeX/texbin/kpsewhich

NB: renamed the executable to avoid conflicts with shell built-ins

[plk]

@herbs - yes we preempted that issue and released a new biblatex version last night just after the cut-off for distribution so you will get 3.18a with the fix today.

For an emergency fix, see:

https://github.com/plk/biblatex/issues/1234

[plk]

@leonmoonen - Aha, that's the issue then. How strange - and that doesn't happen with the raw perl if that's in /Library/TeX/texbin?

[leonmoonen]

nope, the raw perl script in /Library/TeX/texbin gives the complete path in both cases

[plk]

Can you see what happens to the binary and the script when called through the real (no symlinks) absolute path, e.g:

/usr/local/texlive/2022/bin/x86_64-darwinlegacy/biber-test

[leonmoonen]

Through the absolute path, things still break when the executable is found via $PATH, direct calls work as expected

➤ /usr/local/texlive/2022/bin/universal-darwin/biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin:/usr/local/Caskroom/miniconda/base/condabin:/Users/leon/bin:/usr/local/bin:/usr/local/sbin:/usr/local/opt/sqlite/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
/Library/TeX/texbin/kpsewhich

➤ /usr/local/texlive/2022/bin/universal-darwin/raw-biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin:/usr/local/Caskroom/miniconda/base/condabin:/Users/leon/bin:/usr/local/bin:/usr/local/sbin:/usr/local/opt/sqlite/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
/Library/TeX/texbin/kpsewhich

➤ fish_add_path -p /usr/local/texlive/2022/bin/universal-darwin

➤ which biber-test
/usr/local/texlive/2022/bin/universal-darwin/biber-test

➤ biber-test
PATH=/usr/local/texlive/2022/bin/universal-darwin
/usr/local/texlive/2022/bin/universal-darwin/kpsewhich

➤ which raw-biber-test
/usr/local/texlive/2022/bin/universal-darwin/raw-biber-test

➤ raw-biber-test
PATH=/usr/local/Caskroom/miniconda/base/bin:/usr/local/Caskroom/miniconda/base/condabin:/Users/leon/bin:/usr/local/bin:/usr/local/sbin:/usr/local/opt/sqlite/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin
/usr/local/texlive/2022/bin/universal-darwin/kpsewhich

[plk]

I can reproduce this on MacOS 10.12 Intel too. Have opened ticket to PAR::Packer as it seems to be something to do with the binary packaging - see mentioned ticket above.

[plk]

Looks like this might be an issue with an older version of PAR::Packer as I can fix the issue by upgrading this on MacOS 10.12. I will be upgrading the MacOS build servers and pushing new versions soon.