ietf-tools / bibxml-service

Django-based Web service implementing IETF BibXML APIs
https://bib.ietf.org
BSD 3-Clause "New" or "Revised" License
15 stars 19 forks source link

Mapping for IEEE references in `bibxml6` to IEEE dataset #136

Closed ronaldtse closed 2 years ago

ronaldtse commented 2 years ago

This task will fix https://github.com/ietf-ribose/bibxml-service/issues/31

reference.IEEE.730_2014.xml
reference.IEEE.754.1985.xml
reference.IEEE.802-1A.1990.xml
reference.IEEE.802-1D.1990.xml
reference.IEEE.802-1D.1991.xml
reference.IEEE.802-1D.1993.xml
reference.IEEE.802-1G.1992.xml
reference.IEEE.802-1Y.1990.xml
reference.IEEE.802-2.1984.xml
reference.IEEE.802-2.1985.xml
reference.IEEE.802-2.1989.xml
reference.IEEE.802-2.1994.xml
reference.IEEE.802-3.1985.xml
reference.IEEE.802-3.1988.xml
reference.IEEE.802-3.1990.xml
reference.IEEE.802-3.1996.xml
reference.IEEE.802-3.1998.xml
reference.IEEE.802-4.1988.xml
reference.IEEE.802-5.1989.xml
reference.IEEE.802-5.1995.xml
reference.IEEE.802-6.1990.xml
reference.IEEE.802-11.1999.xml
reference.IEEE.802-11A.1999.xml
reference.IEEE.802-11B.1999.xml
reference.IEEE.802-11D.2001.xml
reference.IEEE.802-11F.2003.xml
reference.IEEE.802-11G.2003.xml
reference.IEEE.802-11H.2003.xml
reference.IEEE.802-11I.2004.xml
reference.IEEE.802-12.1995.xml
reference.IEEE.802.1AB_2009.xml
reference.IEEE.802.1AB_2009_COR_1_2013.xml
reference.IEEE.802.1AB_2009_COR_2_2015.xml
reference.IEEE.802.1AC_2012.xml
reference.IEEE.802.1AD_2005.xml
reference.IEEE.802.1AEBN_2011.xml
reference.IEEE.802.1AEBW_2013.xml
reference.IEEE.802.1AE_2006.xml
reference.IEEE.802.1AG_2007.xml
reference.IEEE.802.1AJ_2009.xml
reference.IEEE.802.1AK_2007.xml
reference.IEEE.802.1AP_2008.xml
reference.IEEE.802.1AQ_2012.xml
reference.IEEE.802.1AR_2009.xml
reference.IEEE.802.1AS_2011.xml
reference.IEEE.802.1AS_2011_COR_1_2013.xml
reference.IEEE.802.1AXBK_2012.xml
reference.IEEE.802.1AX_2008.xml
reference.IEEE.802.1AX_2014.xml
reference.IEEE.802.1BA_2011.xml
reference.IEEE.802.1BR_2012.xml
reference.IEEE.802.1D_2004.xml
reference.IEEE.802.1H_1997_EDITION_ISO_IEC_TR11802_5_1997_.xml
reference.IEEE.802.1QAT_2010.xml
reference.IEEE.802.1QAU_2010.xml
reference.IEEE.802.1QAV_2009.xml
reference.IEEE.802.1QAW_2009.xml
reference.IEEE.802.1QAY_2009.xml
reference.IEEE.802.1QAZ_2011.xml
reference.IEEE.802.1QBB_2011.xml
reference.IEEE.802.1QBC_2011.xml
reference.IEEE.802.1QBE_2011.xml
reference.IEEE.802.1QBF_2011.xml
reference.IEEE.802.1QBG_2012.xml
reference.IEEE.802.1QBV_2015.xml
reference.IEEE.802.1QCD_2015.xml
reference.IEEE.802.1Q_2005.xml
reference.IEEE.802.1Q_2005_COR1_2008.xml
reference.IEEE.802.1Q_2011_COR_2_2012.xml
reference.IEEE.802.1Q_2012.xml
reference.IEEE.802.1Q_2014.xml
reference.IEEE.802.1Qcc-2018.xml
reference.IEEE.802.1XBX_2014.xml
reference.IEEE.802.1X_2010.xml
reference.IEEE.802.3.1_2011.xml
reference.IEEE.802.3.1_2013.xml
reference.IEEE.802.3AN_2006.xml
reference.IEEE.802.3AP_2007.xml
reference.IEEE.802.3AQ_2006.xml
reference.IEEE.802.3AS_2006.xml
reference.IEEE.802.3BD_2011.xml
reference.IEEE.802.3BJ_2014.xml
reference.IEEE.802.3BK_2013.xml
reference.IEEE.802.3BM_2015.xml
reference.IEEE.802.3_2012.xml
reference.IEEE.802.5T_2000.xml
reference.IEEE.802.5V_2001.xml
reference.IEEE.802.5W_2000.xml
reference.IEEE.802.11AA_2012.xml
reference.IEEE.802.11AC_2013.xml
reference.IEEE.802.11AD_2012.xml
reference.IEEE.802.11AE_2012.xml
reference.IEEE.802.11AF_2013.xml
reference.IEEE.802.11_2012.xml
reference.IEEE.802.15.1_2005.xml
reference.IEEE.802.15.2_2003.xml
reference.IEEE.802.15.3B_2005.xml
reference.IEEE.802.15.3C_2009.xml
reference.IEEE.802.15.3_2003.xml
reference.IEEE.802.15.4A_2007.xml
reference.IEEE.802.15.4C_2009.xml
reference.IEEE.802.15.4D_2009.xml
reference.IEEE.802.15.4E_2012.xml
reference.IEEE.802.15.4F_2012.xml
reference.IEEE.802.15.4G_2012.xml
reference.IEEE.802.15.4J_2013.xml
reference.IEEE.802.15.4K_2013.xml
reference.IEEE.802.15.4M_2014.xml
reference.IEEE.802.15.4P_2014.xml
reference.IEEE.802.15.4_2011.xml
reference.IEEE.802.15.5_2009.xml
reference.IEEE.802.15.6_2012.xml
reference.IEEE.802.15.7_2011.xml
reference.IEEE.802.16.1A_2013.xml
reference.IEEE.802.16.1B_2012.xml
reference.IEEE.802.16.1_2012.xml
reference.IEEE.802.16.2_2004.xml
reference.IEEE.802.16K_2007.xml
reference.IEEE.802.16M_2011.xml
reference.IEEE.802.16P_2012.xml
reference.IEEE.802.16Q_2015.xml
reference.IEEE.802.16_CONFORMANCE04_2006.xml
reference.IEEE.802.17A_2004.xml
reference.IEEE.802.17B_2007.xml
reference.IEEE.802.17C_2010.xml
reference.IEEE.802.17_2004.xml
reference.IEEE.802.17_2011.xml
reference.IEEE.802.19.1_2014.xml
reference.IEEE.802.20.2_2010.xml
reference.IEEE.802.20.3_2010.xml
reference.IEEE.802.20A_2010.xml
reference.IEEE.802.20B_2010.xml
reference.IEEE.802.20_2008.xml
reference.IEEE.802.21A_2012.xml
reference.IEEE.802.21B_2012.xml
reference.IEEE.802.21C_2014.xml
reference.IEEE.802.21D_2015.xml
reference.IEEE.802.21_2008.xml
reference.IEEE.802.22.1_2010.xml
reference.IEEE.802.22.2_2012.xml
reference.IEEE.802.22A_2014.xml
reference.IEEE.802.22_2011.xml
reference.IEEE.802.1990.xml
reference.IEEE.802A_2003.xml
reference.IEEE.802_2001.xml
reference.IEEE.802_2014.xml
reference.IEEE.828_2012.xml
reference.IEEE.829_2008.xml
reference.IEEE.982.1_2005.xml
reference.IEEE.1003-1G.1997.xml
reference.IEEE.1003-2.1992.xml
reference.IEEE.1003.1_2013_EDITION.xml
reference.IEEE.1003.13_2003.xml
reference.IEEE.1003.26_2003.xml
reference.IEEE.1008_1987.xml
reference.IEEE.1012_2012.xml
reference.IEEE.1016_2009.xml
reference.IEEE.1028_2008.xml
reference.IEEE.1044_2009.xml
reference.IEEE.1061_1998.xml
reference.IEEE.1062_1998.xml
reference.IEEE.1074_2006.xml
reference.IEEE.1175.1_2002.xml
reference.IEEE.1175.2_2006.xml
reference.IEEE.1175.3_2004.xml
reference.IEEE.1175.4_2008.xml
reference.IEEE.1220_2005.xml
reference.IEEE.1228_1994.xml
reference.IEEE.1278-1.1995.xml
reference.IEEE.1278-2.1995.xml
reference.IEEE.1284-1.1997.xml
reference.IEEE.1320.1_1998.xml
reference.IEEE.1320.2_1998.xml
reference.IEEE.1394.1995.xml
reference.IEEE.1462_1998.xml
reference.IEEE.1490_2011.xml
reference.IEEE.1517_2010.xml
reference.IEEE.1633_2008.xml
reference.IEEE.1636.1_2013.xml
reference.IEEE.1636.99_2013.xml
reference.IEEE.1647_2011.xml
reference.IEEE.1874_2013.xml
reference.IEEE.2001_2002.xml
reference.IEEE.3333.1.1_2015.xml
reference.IEEE.8802.1AR_2014.xml
reference.IEEE.8802_1AB_2014.xml
reference.IEEE.8802_1AE_2013.xml
reference.IEEE.8802_1AS_2014.xml
reference.IEEE.8802_1X_2013.xml
reference.IEEE.8802_3_2014.xml
reference.IEEE.8802_5_1998.xml
reference.IEEE.8802_11_2012.xml
reference.IEEE.8802_11_2012_AMD.1_2014.xml
reference.IEEE.8802_11_2012_AMD.2_2014.xml
reference.IEEE.8802_11_2012_AMD.3_2014.xml
reference.IEEE.8802_22_2015.xml
reference.IEEE.12207_2008.xml
reference.IEEE.13210_1994.xml
reference.IEEE.14102_2010.xml
reference.IEEE.14143.1_2000.xml
reference.IEEE.14471_2010.xml
reference.IEEE.14515_1_2000.xml
reference.IEEE.15026_1_2014.xml
reference.IEEE.15026_2_2011.xml
reference.IEEE.15026_3_2013.xml
reference.IEEE.15026_4_2013.xml
reference.IEEE.15288.1_2014.xml
reference.IEEE.15288.2_2014.xml
reference.IEEE.15288_2008.xml
reference.IEEE.15288_2015.xml
reference.IEEE.15289_2015.xml
reference.IEEE.15939_2008.xml
reference.IEEE.16085_2006.xml
reference.IEEE.16326_2009.xml
reference.IEEE.20000_1_2013.xml
reference.IEEE.20000_2_2013.xml
reference.IEEE.23026_2015.xml
reference.IEEE.24748_1_2011.xml
reference.IEEE.24748_2_2012.xml
reference.IEEE.24748_3_2012.xml
reference.IEEE.24765_2010.xml
reference.IEEE.24774_2012.xml
reference.IEEE.26511_2012.xml
reference.IEEE.26512_2011.xml
reference.IEEE.26513_2010.xml
reference.IEEE.26514_2010.xml
reference.IEEE.26515_2012.xml
reference.IEEE.26531_2015.xml
reference.IEEE.29119_1_2013.xml
reference.IEEE.29119_2_2013.xml
reference.IEEE.29119_3_2013.xml
reference.IEEE.29148_2011.xml
reference.IEEE.31320_1_2012.xml
reference.IEEE.31320_2_2012.xml
reference.IEEE.42010_2011.xml
reference.IEEE.90003_2008.xml
reference.IEEE.ISO_IEC_14515_1_2000_2003.1_2000.xml
reference.IEEE.ISO_IEC_14519_2001_1003.5_1999.xml
reference.IEEE.ISO_IEC_14764_IEEE_STD_14764_2006.xml
reference.IEEE.ISO_IEC_26702_IEEE_STD_1220_2005_FIRST_EDITION_2007_07_15.xml
reference.IEEE.ISO_IEC_42010_IEEE_STD_1471_2000_FIRST_EDITION_2007_07_15.xml
reference.IEEE.ISO_IEC_IEEE_9945.xml
reference.IEEE.P802-1A.1989.xml
reference.IEEE.P802-1Q.1998.xml
reference.IEEE.P802-3K.1992.xml
reference.IEEE.P802-3P.1992.xml
reference.IEEE.P802-5D.1989.xml
reference.IEEE.P802.11U.xml
reference.IEEE.P1363.1998.xml
reference.IEEE.P1394A.1995.xml
reference.IEEE.P1394B.1995.xml
reference.IEEE.P8021A.1989.xml
reference.IEEE.P8021D.1989.xml
ronaldtse commented 2 years ago

Here's the mapping. There are entries that are marked "UNKNOWN" or "NOT AVAILABLE IN DATASET".

ieee-mapping.yaml.zip

Warning that some of the "NOT AVAILABLE IN DATASET" entries are due to https://github.com/relaton/relaton-ieee/issues/16, so they may become available once we are able to parse all IEEE entries.

ronaldtse commented 2 years ago

This is the new set of mappings from bibxml6 filenames to IEEE dataset's title attribute. Only the following items are missing.

ieee-mapping.yaml.zip

These are the document identifiers that we cannot find in the IEEE dataset: not available on ieee.org and cannot be found anywhere (even when relaton/relaton-ieee#16 is complete). We need to find a way to resolve these. The last two possibly point to ISO co-published copies, but IEEE's dataset doesn't contain them.

reference.IEEE.802-1D.1991.xml: UNKNOWN; ONLY INTERNET SOURCE IS bibxml2
reference.IEEE.802-1D.1993.xml: UNKNOWN; ONLY INTERNET SOURCE IS bibxml2
reference.IEEE.802-1Y.1990.xml: UNKNOWN; there is a 802.1Y but is a completely different document.
reference.IEEE.P802-1A.1989.xml: UNKNOWN, ONLY INTERNET SOURCE IS bibxml2
reference.IEEE.P8021A.1989.xml: UNKNOWN, ONLY INTERNET SOURCE IS bibxml2
reference.IEEE.802-3.1990.xml: DOES NOT EXIST IN IEEE DATASET - ISO/IEC 8802-3:1990
reference.IEEE.802-3.1996.xml: DOES NOT EXIST IN IEEE DATASET - ISO/IEC 8802-3:1996
ronaldtse commented 2 years ago

I've investigated these last 7 entries:

I have also verified that all bibxml2/reference.IEEE.* entries exist in bibxml6/.

The recommendations are:

@rjsparks we will need guidance on next steps here. Thanks!

ronaldtse commented 2 years ago

(Once this task is done we can remove all bibxml2/reference.IEEE.* files from the bibxml-data-archive repo.)

rjsparks commented 2 years ago

@TonyLHansen Comments on the choice in the last bullet above?

russhousley commented 2 years ago

I do not think we should deviate away from the IEEE dataset. This will keep it simple when IEEE releases updates.

Any supplemental entries can be put in bibxml-misc.

strogonoff commented 2 years ago

Here's the mapping. There are entries that are marked "UNKNOWN" or "NOT AVAILABLE IN DATASET".

ieee-mapping.yaml.zip

These mappings seem to reference many document identifiers not present in IEEE bibliography source🤔

and others.

I’m not sure whether it has to do with wrong identifier format, or documents are missing altogether.

strogonoff commented 2 years ago

Answered my own question.

Mappings for bibxml6 were added to bibxml-data-archive in https://github.com/ietf-ribose/bibxml-data-archive/commit/c6b1281a13add3bc32ed229847f2162b737af3ef. Most reference nonexistent docids and so will fall back to xml2rfc archive data.

ronaldtse commented 2 years ago

These mappings seem to reference many document identifiers not present in IEEE bibliography source🤔

  • IEEE Std 1062, 1998 Edition
  • IEEE Std 1061-1998
  • IEEE Std 1012-2012 (Revision of IEEE Std 1012-2004) - Redline
  • IEEE Std 1003.1, 2013 Edition (incorporates IEEE Std 1003.1-2008, and IEEE Std 1003.1-2008/Cor 1-2013)

and others.

I’m not sure whether it has to do with wrong identifier format, or documents are missing altogether.

We actually need to fix the relaton-data-ieee document identifiers to show the correct format. It is time to integrate relaton-ieee with pubid-ieee.

strogonoff commented 2 years ago

I’m happy if we do that, but I’m not sure if this will address the invalid mappings in short term…