Closed logological closed 5 years ago
@villalbamartin, I can confirm this is correct. The original bib file for D18-1350, for example, has the month field, and this is preserved in the XML, but it is not present in the regenerated bib file.
How hard would this be to add the month field to the DB export?
Hi Matt, all:
It'd be pretty easy to do, and an #enhancement that you might want to
prioritize at some point, either through Github Projects, or Issue tags or
milestones.
Softconf also now can export
Cheers,
Min
-- Min-Yen KAN (Dr) :: Associate Professor :: National University of Singapore :: NUS School of Computing, AS6 05-12, 13 Computing Drive Singapore 117417 :: +65 6516 1885(DID) :: +65 6779 4580 (Fax) :: kanmy@comp.nus.edu.sg (E) :: www.comp.nus.edu.sg/~kanmy (W)
On Fri, Jan 4, 2019 at 11:40 PM Matt Post notifications@github.com wrote:
@villalbamartin https://github.com/villalbamartin, I can confirm this is correct. The original bib file for D18-1350 https://aclweb.org/anthology/D18-1350.bib, for example, has the month field, and this is preserved in the XML https://github.com/acl-org/acl-anthology/blob/b6eea2b2f1a00f046ee9ae20aa2cc245cd5abfd6/import/D18.xml#L5992, but it is not present in the regenerated bib file https://aclanthology.coli.uni-saarland.de/papers/D18-1350/d18-1350.bib.
How hard would this be to add the month field to the DB export?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/acl-org/acl-anthology/issues/94#issuecomment-451479025, or mute the thread https://github.com/notifications/unsubscribe-auth/AANP62EWgd5dwV1yfatUO4ZmoXKzII6Rks5u_3XYgaJpZM4YU1wr .
I've looked at the code. The file responsible for the bibtex export function is lib/tasks/bib_export.rake, which first creates an XML file and then uses xml2bib
to generate the .bib file (seen here: https://github.com/acl-org/acl-anthology/blob/b6eea2b2f1a00f046ee9ae20aa2cc245cd5abfd6/lib/tasks/bib_export.rake#L288)
I'm not familiar with rexml
(which I believe is what the code uses to generate the XML structure to begin with), but I'm willing to believe that it shouldn't be that difficult.
The trickiest part is actually pulling the month from the database. I've run the query SELECT DISTINCT month FROM papers ORDER BY month ASC
, and got the following results:
So that part may need to be hand-coded.
Thanks for this information! It seems the month format is not consistent across entries. But why does this matter? When generating the .bib
file for each paper, can't you just use whatever is listed for that paper in the database?
Thanks for this information! It seems the month format is not consistent across entries. But why does this matter? When generating the
.bib
file for each paper, can't you just use whatever is listed for that paper in the database?
You can, but failure to use the standard BibTeX month macros will be a perpetual annoyance to those who want to use our BibTeX entries in non-English documents. (Or even in English ones if the bibliography style formats months differently than whatever fixed string the database uses. For example, some bibliography styles mandate abbreviated month names like "Sept." instead of "September". BibTeX will handle this automatically when the value of the month field is sep
, but not when it is {September}
or "September"
.)
Agreed, but these are two separate issues: (a) properly generating the BibTeX files from our authoritative repository and (b) ensuring that the authoritative repository is correct. For (a), we can use a quick-fix of just exporting the data that is already there. Going into the future, we should both (b.1) fix the ingest procedures to enforce correctness (#90; https://github.com/acl-org/acl-pub/issues/9) and, longer-term (b.2) fix the current data.
So the question is one of correctness versus completeness---whether it is worthwhile to output the existing month field as-is from the database as a short-term fix, or whether we should wait until the data is fixed. I'm inclined towards the former.
There are also other scattered issues, e.g., hyphens where there should be n-dashes in ranges.
I wrote a patch for this - I'm testing it now on the test server, but it will take ~12hs to finish. Here's my proposed solution:
month
field only can contain one month, the only other alternative is to not include month for publications with a month in this format.Once I've checked that the current code works, I'll publish a pull request.
Hi Martín, all:
For one month cases, it should be ok.
For those with more than one month, probably better to stay with just having the current string, as-is.
For the special case where assumedly a date is there "7-8" June, sounds like that was a coding error. We can remove the date and code "June".
What do you think?
On Sun, Jan 6, 2019 at 3:09 AM Martín Villalba notifications@github.com wrote:
I wrote a patch for this - I'm testing it now on the test server, but it will take ~12hs to finish. Here's my proposed solution:
If the month is not available, we simply output the year If the month is available, we take the first 3 lowercase letters and turn this into a month. In this way, a case like "September/December" is assumed to be September. Given that the month field only can contain one month, the only other alternative is to not include month for publications with a month in this format. The special case "7-8 June" is handled separately, since it's the only field that does not do anything reasonable with the heuristic from the previous point For any other kind of improper input, we only output the year
Once I've checked that the current code works, I'll publish a pull request.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.
Hi Min,
here's a long description of the challenges, mostly for future reference. Spoiler: we can do it, but it's not as straightforward as simply passing a string.
There are two issues to consider if we want to use a multi-month format, one of "pipeline" and one of what Bibtex parsers expect.
From a pipeline point of view, the current story goes like this:
year
and month
fields are retrieved as-is from the databasedateIssued
, which expects a field with format yyyy[-mm]. This information is dumped to a MODS XML file.xml2bib
utility with the XML file as input.Therefore, if we want a multi-month entry we would have to change the format of dateIssued
. According to the specification we can extend the dateIssued
field with start
and end
attributes. In that case, we delegate to the utilities how that field is handled, but my tests reveal that xml2bib
ignores this attribute and simply chooses the first date in the file. We would therefore have to write code to tweak the output file, generating a standard file first and modifying its month
field later. What to put in there is the other point that we need to consider.
According to "LaTeX: A document preparation system, User’s guide and reference manual", Appendix B1.3, the month
field should contain a three-letter abbreviation of a month which is later translated to the user's language, so our implementation would not be up to standard. This is not a big issue, since I've noted that most .bib files choose to ignore the standard anyway. If we want to be nice for people using LaTeX in languages other than English, we should use this workaround:
month = jun # "\slash " # jul,
Doing the same for the other formats (Word, ENDF, etc.) would require modifying files according to their own formats, but I'm not really sure how many people actually use those.
For those with more than one month, probably better to stay with just having the current string, as-is.
For month ranges, the "official" way of doing this is still to use the three-letter macros. In this case, string concatenation can be used. So month = {January--February}
should actually be month = jan # "--" # feb
.
Thanks, all. @villalbamartin, a PR will be a great way to continue conversation.
@mjpost I've done that now. As a side note, issue #107 was created because I was worried that the function date_formatter
I wrote would misbehave for strange input. Luckily, it seems all of our input is well-behaved.
My recent commit introduced the three-letter macros for the month field. Can someone double-check if there's anything left to be done on this, or if it can be closed?
This looks good to me. I parsed the whole anthology.bib.gz file with Pybtex and it loaded fine, and entries for conferences that span the month boundary also look good (e.g., N15-4002).
Thanks again to @logological for providing detailed information on the BibTeX spec!
The HTML publication records in the Anthology include a month field, but this field does not appear in the BibTeX export. Please consider including this field in the BibTeX record. Note that, wherever possible, the value of this field should be a standard BibTeX month macro (
jan
,feb
,mar
, etc.) and not a string ({January}
,"February"
, etc.) as this allows bibliographies to be localized by the BibTeX style.