search

BeelGroup / Docear-Desktop

Docear's desktop version (GPL)
298 stars 90 forks source link

More robust BibTeX parsing for Zotero and Mendeley #416

Open Joeran opened 10 years ago

Joeran commented 10 years ago

One user exported his library from Zotero which contained this entry

@article{horn_spectral_1986,
    title = {Spectral transmission characteristics of shocked optical fibers},
    volume = {25},
    url = {http://ao.osa.org/abstract.cfm?URI=ao-25-19-3430},
    doi = {10.1364/AO.25.003430},
    number = {19},
    urldate = {2013-07-31},
    journal = {Applied Optics},
    author = {Horn, Paul D. and Gupta, Yogendra M.},
    month = oct,
    year = {1986},
    pages = {3430--3434},
Axial impact of silica and sapphire fibers (with horizontal constraint).},
}

The last line is invalid and causes Docear to not load the BibTeX file at all. Ideally, the entry should just be ignored or an error message should appear. Actually, JabRef shows an error message when the BibTeX file is opened, but Docear does not image

Joeran commented 10 years ago

see also http://www.docear.org/support/forums/docear-support-forums-group3/bug-reports-forum6/reference-pane-emtpy-and-greyed-out-after-update-thread945.0/

Joeran commented 10 years ago

It seems that (sometimes) Docear itself is causing the problem, i.e. Docear is parsing a BibTeX file incorrectly and corrupting it.

For an example file, see the email "Docear seems to corrupt Bibtex file from Zotero".

The original file contains, among others, this entry:

@misc{_amf_????,
    title = {{AMF} - Advanced Motor Fuels},
    url = {http://www.iea-amf.org/content/fuel_information/methanol},
    urldate = {2014-10-03},
    annote = {top site van het interntionaal energie agentschap voor info! methanol ethanol etz
 }
}

The file can be opened with JabRef and Docear4Word. However, when opening it with Docear, Docear a) does not display the file and b) changes the entry to

@misc{_amf_????,
    title = {{AMF} - Advanced Motor Fuels},
    url = {http://www.iea-amf.org/content/fuel_information/methanol},
    urldate = {2014-10-03},
}

Here, another example of an entry that works in JabRef and Docear4Word:

@article{qi_properties_2005,
    title = {Properties, performance, and emissions of methanol-gasoline blends in a spark ignition engine},
    volume = {219},
    copyright = {Copyright Mechanical Engineering Publications, Ltd. Mar 2005},
    issn = {09544070},
    url = {http://search.proquest.com/docview/220660789?accountid=11077},
    abstract = {One of the major problems for the successful application of a methanol-gasoline blend as a motor fuel was the realization of a stable homogeneous liquid phase. This paper studied the effect of ethanol as the co-solvent in the methanol-gasoline blend in order to overcome this problem. In this way, not only was the phase separation problem solved but the methanol ratio in the blend was also increased. The critical phase separation temperature ({CPST}) of the methanol-gasoline blend increased with increasing water content in the blend, and the addition of ethanol caused the {CPST} to decrease. M10 (gasoline containing 8.5 vol \% methanol and 1.5 vol \% ethanol) and M25 (gasoline containing 19 vol \% methanol and 6 vol \% ethanol) were exploited to test the performance, the fuel consumption, and the exhaust emissions. The results show that the specific fuel consumption of M10 was almost the same as that of gasoline, but that of M25 was higher for all engine speeds at full load. The specific energy consumption of gasoline was higher than that of blends for all engine speeds at full load and that of M25 was lower under low load at a fixed engine speed. The engine torque and power output were observed to be lower than those of gasoline, and it was found that the higher the volume fraction of methanol in blend, the larger the reduction. The hydrocarbon emission concentration of M25 was higher and the nitrogen oxides emission concentration was lower than those of gasoline and M10 for all engine loads. Under low and moderate loads, the carbon monoxide concentration of gasoline was higher than that of methanol-gasoline blends, but under high loads that of M25 was higher. [{PUBLICATION} {ABSTRACT}]
Keywords: methanol-gasoline blends, spark ignition engine, ethanol, critical phase separation temperature ({CPST}), exhaust emissions},
    language = {English},
    number = {3},
    urldate = {2014-09-28},
    journal = {Proceedings of the Institution of Mechanical Engineers},
    author = {Qi, D. H. and Liu, Sh Q. and Liu, J. C. and Zhang, Ch H. and Bian, Y. Zh},
    month = mar,
    year = {2005},
    keywords = {Automobile engines, Engineering--Mechanical Engineering, ethanol, Mechanical engineering, Reformulated gasoline, Vehicle emissions},
    pages = {405--412},
    annote = {{QI} D
 
gives data on methanol/gasoline mixability
add ethanol for good measure
 },
    file = {Qi et al. - 2005 - Properties, performance, and emissions of methanol.pdf:C\:\\Zotero\\storage\\storage\\ETAB9GWE\\Qi et al. - 2005 - Properties, performance, and emissions of methanol.pdf:application/pdf}
}

And after opening it with Docear:

@article{qi_properties_2005,
    title = {Properties, performance, and emissions of methanol-gasoline blends in a spark ignition engine},
    volume = {219},
    copyright = {Copyright Mechanical Engineering Publications, Ltd. Mar 2005},
    issn = {09544070},
    url = {http://search.proquest.com/docview/220660789?accountid=11077},
    abstract = {One of the major problems for the successful application of a methanol-gasoline blend as a motor fuel was the realization of a stable homogeneous liquid phase. This paper studied the effect of ethanol as the co-solvent in the methanol-gasoline blend in order to overcome this problem. In this way, not only was the phase separation problem solved but the methanol ratio in the blend was also increased. The critical phase separation temperature ({CPST}) of the methanol-gasoline blend increased with increasing water content in the blend, and the addition of ethanol caused the {CPST} to decrease. M10 (gasoline containing 8.5 vol \% methanol and 1.5 vol \% ethanol) and M25 (gasoline containing 19 vol \% methanol and 6 vol \% ethanol) were exploited to test the performance, the fuel consumption, and the exhaust emissions. The results show that the specific fuel consumption of M10 was almost the same as that of gasoline, but that of M25 was higher for all engine speeds at full load. The specific energy consumption of gasoline was higher than that of blends for all engine speeds at full load and that of M25 was lower under low load at a fixed engine speed. The engine torque and power output were observed to be lower than those of gasoline, and it was found that the higher the volume fraction of methanol in blend, the larger the reduction. The hydrocarbon emission concentration of M25 was higher and the nitrogen oxides emission concentration was lower than those of gasoline and M10 for all engine loads. Under low and moderate loads, the carbon monoxide concentration of gasoline was higher than that of methanol-gasoline blends, but under high loads that of M25 was higher. [{PUBLICATION} {ABSTRACT}]
Keywords: methanol-gasoline blends, spark ignition engine, ethanol, critical phase separation temperature ({CPST}), exhaust emissions},
    language = {English},
    number = {3},
    urldate = {2014-09-28},
    journal = {Proceedings of the Institution of Mechanical Engineers},
    author = {Qi, D. H. and Liu, Sh Q. and Liu, J. C. and Zhang, Ch H. and Bian, Y. Zh},
    month = mar,
    year = {2005},
    keywords = {Automobile engines, Engineering--Mechanical Engineering, ethanol, Mechanical engineering, Reformulated gasoline, Vehicle emissions},
    pages = {405--412},
    file = {Qi et al. - 2005 - Properties, performance, and emissions of methanol.pdf:C\:\\Zotero\\storage\\storage\\ETAB9GWE\\Qi et al. - 2005 - Properties, performance, and emissions of methanol.pdf:application/pdf}
}

I assume that the line breaks in the annote field cause the problem (the closing bracket is in the next line). Although, the changes I found do not explain why Docear is not displaying the file at all

Joeran commented 9 years ago

A Mendeley user reports the same problem. When Docear opens the file, Docear corrupts it. From originally 3 MB only 800kb remain. To get the BibTeX file, search in our mail archive for "Help with migrating to Docear"

After creating a new project in Docear, and then changing the reference database from the default file to Mendeley's BibTeX, these exceptions occur. 

INFO: Loaded properties from bundle://2.0:0/org/docear/plugin/bibtex/defaults.properties
Nov 13, 2014 8:53:13 AM net.sf.jabref.plugin.PluginCore initialize
WARNING: Error in starting plug-in system. Starting without, but some functionality may be missing.
java.io.File

STDOUT: JabRef exception suppressed by Docear: nullNov 13, 2014 8:53:14 AM org.freeplane.core.util.LogUtils info
INFO: class org.freeplane.plugin.workspace.WorkspaceController.addAction(): action workspace.action.project.new not added! (action workspace.action.project.new already registered)
Nov 13, 2014 8:53:14 AM org.freeplane.core.util.LogUtils info
INFO: class org.freeplane.plugin.workspace.WorkspaceController.addAction(): action workspace.action.project.import not added! (action workspace.action.project.import already registered)
Nov 13, 2014 8:53:14 AM org.freeplane.core.util.LogUtils info
INFO: class org.freeplane.plugin.workspace.WorkspaceController.addAction(): action workspace.action.node.open.location not added! (action workspace.action.node.open.location already registered)
Nov 13, 2014 8:53:14 AM org.freeplane.core.util.LogUtils info
INFO: DOCEAR: adding new mindmap to library: file:/C:/Users/admin/Docear/projects/My%20Thesis%2010/_data/149A4A74ECC956VQIARCNXBUSCW7A0AENQ3S/default_files/literature_and_annotations.mm
Nov 13, 2014 8:53:14 AM org.freeplane.core.util.LogUtils info
INFO: DOCEAR: adding new mindmap to library: file:/C:/Users/admin/Docear/projects/My%20Thesis%2010/_data/149A4A74ECC956VQIARCNXBUSCW7A0AENQ3S/default_files/temp.mm
Nov 13, 2014 8:53:14 AM org.freeplane.core.util.LogUtils info
INFO: DOCEAR: adding new mindmap to library: file:/C:/Users/admin/Docear/projects/My%20Thesis%2010/_data/149A4A74ECC956VQIARCNXBUSCW7A0AENQ3S/default_files/trash.mm
Nov 13, 2014 8:53:14 AM org.freeplane.core.util.LogUtils info
INFO: [org.freeplane.plugin.workspace.creator.FolderFileNodeCreator] addFileType: directory_handle
Nov 13, 2014 8:53:14 AM org.freeplane.core.util.LogUtils info
INFO: [org.freeplane.plugin.workspace.creator.DefaultFileNodeCreator] addFileType: default_handle
Nov 13, 2014 8:53:14 AM org.freeplane.core.util.LogUtils info
INFO: project My Thesis 10 loaded in: 82
Nov 13, 2014 8:53:14 AM org.freeplane.core.util.LogUtils info
INFO: Opening References: 'C:\Users\admin\Docear\projects\My Thesis 10\_data\149A4A74ECC956VQIARCNXBUSCW7A0AENQ3S\default_files\My Thesis 10.bib'

STDOUT: class org.docear.plugin.bibtex.jabref.DocearTransformForeignDatabaseAction time: 0ms
STDOUT: class org.docear.plugin.bibtex.actions.FilePathValidatorAction time: 0msNov 13, 2014 8:53:14 AM org.freeplane.core.util.LogUtils info
INFO: Opened database 'C:\Users\admin\Docear\projects\My Thesis 10\_data\149A4A74ECC956VQIARCNXBUSCW7A0AENQ3S\default_files\My Thesis 10.bib' with 4 entries.
Nov 13, 2014 8:53:14 AM org.freeplane.core.util.LogUtils info
INFO: database C:\Users\admin\Docear\projects\My Thesis 10\_data\149A4A74ECC956VQIARCNXBUSCW7A0AENQ3S\default_files\My Thesis 10.bib loaded in: 306

STDOUT: Framework launched
STDOUT: REMINDERHOOK: org.freeplane.features.mode.mindmapmode.MModeController@7e6a31e1Nov 13, 2014 8:53:15 AM org.freeplane.core.util.LogUtils info
INFO: requesting mode: MindMap
Nov 13, 2014 8:53:20 AM org.freeplane.core.util.LogUtils info
INFO: menu items to execute: []
Nov 13, 2014 8:55:24 AM org.freeplane.core.util.LogUtils info
INFO: Opening References: 'C:\Users\admin\Downloads\library.bib'

STDERR: java.lang.RuntimeException: Error in line 360: Expected = but received t
STDERR:     at net.sf.jabref.imports.BibtexParser.consume(BibtexParser.java:976)
STDERR:     at net.sf.jabref.imports.BibtexParser.parseField(BibtexParser.java:514)
STDERR:     at net.sf.jabref.imports.BibtexParser.parseEntry(BibtexParser.java:503)
STDERR:     at net.sf.jabref.imports.BibtexParser.parse(BibtexParser.java:367)
STDERR:     at net.sf.jabref.imports.OpenDatabaseAction.loadDatabase(OpenDatabaseAction.java:473)
STDERR:     at net.sf.jabref.imports.OpenDatabaseAction.loadDataBase(OpenDatabaseAction.java:326)
STDERR:     at org.docear.plugin.bibtex.jabref.JabrefWrapper.openIt(JabrefWrapper.java:352)
STDERR:     at org.docear.plugin.bibtex.jabref.JabrefWrapper.openDatabase(JabrefWrapper.java:205)
STDERR:     at org.docear.plugin.bibtex.ReferencesController$13.run(ReferencesController.java:579)
STDERR:     at java.awt.event.InvocationEvent.dispatch(Unknown Source)
STDERR:     at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
STDERR:     at java.awt.EventQueue.access$200(Unknown Source)
STDERR:     at java.awt.EventQueue$3.run(Unknown Source)
STDERR:     at java.awt.EventQueue$3.run(Unknown Source)
STDERR:     at java.security.AccessController.doPrivileged(Native Method)
STDERR:     at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
STDERR:     at java.awt.EventQueue.dispatchEvent(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.run(Unknown Source)Nov 13, 2014 8:55:24 AM org.freeplane.core.util.LogUtils warn
WARNING: ERROR: Could not load fileC:\Users\admin\Downloads\library.bib

STDERR: java.lang.RuntimeException: Error in line 360: Expected = but received t
STDERR:     at net.sf.jabref.imports.BibtexParser.consume(BibtexParser.java:976)
STDERR:     at net.sf.jabref.imports.BibtexParser.parseField(BibtexParser.java:514)
STDERR:     at net.sf.jabref.imports.BibtexParser.parseEntry(BibtexParser.java:503)
STDERR:     at net.sf.jabref.imports.BibtexParser.parse(BibtexParser.java:367)
STDERR:     at net.sf.jabref.imports.OpenDatabaseAction.loadDatabase(OpenDatabaseAction.java:473)
STDERR:     at net.sf.jabref.imports.OpenDatabaseAction.loadDataBase(OpenDatabaseAction.java:326)
STDERR:     at org.docear.plugin.bibtex.jabref.JabrefWrapper.openIt(JabrefWrapper.java:352)
STDERR:     at org.docear.plugin.bibtex.jabref.JabrefWrapper.openDatabase(JabrefWrapper.java:205)
STDERR:     at org.docear.plugin.bibtex.ReferencesController$13.run(ReferencesController.java:579)
STDERR:     at java.awt.event.InvocationEvent.dispatch(Unknown Source)
STDERR:     at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
STDERR:     at java.awt.EventQueue.access$200(Unknown Source)
STDERR:     at java.awt.EventQueue$3.run(Unknown Source)
STDERR:     at java.awt.EventQueue$3.run(Unknown Source)
STDERR:     at java.security.AccessController.doPrivileged(Native Method)
STDERR:     at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
STDERR:     at java.awt.EventQueue.dispatchEvent(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.run(Unknown Source)Nov 13, 2014 8:55:24 AM org.freeplane.core.util.LogUtils warn
WARNING: ERROR: Could not load fileC:\Users\admin\Downloads\library.bib

STDERR: java.lang.RuntimeException: Error in line 360: Expected = but received t
STDERR:     at net.sf.jabref.imports.BibtexParser.consume(BibtexParser.java:976)
STDERR:     at net.sf.jabref.imports.BibtexParser.parseField(BibtexParser.java:514)
STDERR:     at net.sf.jabref.imports.BibtexParser.parseEntry(BibtexParser.java:503)
STDERR:     at net.sf.jabref.imports.BibtexParser.parse(BibtexParser.java:367)
STDERR:     at net.sf.jabref.imports.OpenDatabaseAction.loadDatabase(OpenDatabaseAction.java:473)
STDERR:     at net.sf.jabref.imports.OpenDatabaseAction.loadDataBase(OpenDatabaseAction.java:326)
STDERR:     at org.docear.plugin.bibtex.jabref.JabrefWrapper.openIt(JabrefWrapper.java:352)
STDERR:     at org.docear.plugin.bibtex.jabref.JabrefWrapper.openDatabase(JabrefWrapper.java:205)
STDERR:     at org.docear.plugin.bibtex.ReferencesController$13.run(ReferencesController.java:579)
STDERR:     at java.awt.event.InvocationEvent.dispatch(Unknown Source)
STDERR:     at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
STDERR:     at java.awt.EventQueue.access$200(Unknown Source)
STDERR:     at java.awt.EventQueue$3.run(Unknown Source)
STDERR:     at java.awt.EventQueue$3.run(Unknown Source)
STDERR:     at java.security.AccessController.doPrivileged(Native Method)
STDERR:     at java.security.ProtectionDomain$1.doIntersectionPrivilege(Unknown Source)
STDERR:     at java.awt.EventQueue.dispatchEvent(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
STDERR:     at java.awt.EventDispatchThread.run(Unknown Source)Nov 13, 2014 8:55:24 AM org.freeplane.core.util.LogUtils warn
WARNING: ERROR: Could not load fileC:\Users\admin\Downloads\library.bib
Joeran commented 9 years ago

workaround for all users having this problem: delete your "annotation" fields.