Stange characters when marking No DOI found is exported

[jason-neal]

I ran the clean and verify command on a bunch of references. I was going to commit my exported .bib file with my document when I noticed that the keywords have a bunch of strange characters before No DOI found. I later see that it is the magnifying glass image.

e.g. keywords = {🔍No DOI found}, or keywords = {Astrophysics,🔍No DOI found},

keywords = {⚠️Invalid DOI},

in text the three question mark characters are \x9f \x94 \x8d (without spaces)

It would be nice to enable these to be cleaned up somehow. Either keeping them in the gui and removing when exporting, or a toggle option to include?

Thanks.

[jason-neal]

A exported .bib file with these characters cannot be pushed to overleaf. Overleaf rejects the push due to the .bib file with the magnifying glass characters. This was confusing to find the issue.

$ git push overleaf master 
Counting objects: 26, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (26/26), done.
Writing objects: 100% (26/26), 38.79 KiB | 7.76 MiB/s, done.
Total 26 (delta 16), reused 0 (delta 0)
remote: Resolving deltas: 100% (16/16)
remote: error: invalid files
remote: 
remote: hint: You have 1 invalid files in your Overleaf project:
remote: hint: main.bib (error)
To https://git.overleaf.com/XXXXXXXXXXXXXXX
 ! [remote rejected] master -> master (invalid files)
error: failed to push some refs to 'https://git.overleaf.com/XXXXXXXXXXXx'

[bwiernik]

The magnifying glass is valid Unicode. How exactly are you exporting the items?

[jason-neal]

Right click on a collection, click Export Collection then with Format=Better Bibtex click OK and choose location to save it.

If I open the bib file then I do see the magnifying glass in pluma and texstudio.

With git gui it shows up as

If I stage the whole bib file then it is "happy" but if I selectively stage only this line/hunk then it shows up again as being changed. In a terminal (zsh) at this stage with this line only selectively committed git diff results in

If I commit whole file and try and push it to overleaf I get

git push overleaf master
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 469 bytes | 469.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2)
remote: error: invalid files
remote: 
remote: hint: You have 1 invalid files in your Overleaf project:
remote: hint: thesis.bib (error)
To https://git.overleaf.com/XXXXXXXXXXXXXXXXX
 ! [remote rejected] master -> master (invalid files)
error: failed to push some refs to 'https://git.overleaf.com/XXXXXXXXXXXXXXXXX'

If I remove the line and amend the commit from the bib file and

~/Phd/Writing-in-Progress/thesis master* ⇡
❯ git commit --amend
[master 76ebfd1] Update bib
 Date: Mon Aug 13 11:06:25 2018 +0100
 1 file changed, 3 insertions(+), 3 deletions(-)

~/Phd/Writing-in-Progress/thesis master* ⇡ 7s
❯ git push overleaf master
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 428 bytes | 428.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Resolving deltas: 100% (2/2)
remote: Updating references: 100% (1/1)
To https://git.overleaf.com/XXXXXXXXXXXXXX
   03e0c4c..76ebfd1  master -> master

[bwiernik]

Check that you have the "Export unicode as plain-text latex commands" box check in the BBT preferences. Otherwise, it appears that whatever program you are using to stage your files isn't reading the exported file with the right text encoding. Ensure that it is using UTF-8 throughout the workflow.

I can add an option to use ASCII characters to label the the tags, but I won't have time to do that for a few weeks. In the mean time, you can rename the tag in Zotero before exporting or using find-and-replace (e.g., by a batch file or shell script) before staging the diff to workaround.

[jason-neal]

"Export unicode as plain-text latex commands" was checked. And I am pretty sure ´git gui´ is using utf-8 so I am not sure where it is occurring.

An ASCII option would be good, although there is no rush for me, for now I have already removed the labels.

[bwiernik]

Customizable tags added in https://github.com/bwiernik/zotero-shortdoi/commit/a1a0f2b73d51d4b33d9b7059f77dad21fa9f6aa2