search

ess-acppo / ag-bie

This repository has the code to the agriculture's implementation of ALA BIE
1 stars 4 forks source link

kingdom is null #27

Closed mbohun closed 4 years ago

mbohun commented 5 years ago

While implementing the ROSCO agbiePlugin.php I noticed a JSON record with kingdom set to null: https://ag-bie.oztaxa.com/ws/search.json?q=%22Acacia%20nilotica%22

{
  "searchResults": {
    "totalRecords": 11,
    "facetResults": [],
    "results": [
      {
        "id": "6831e92d-c099-4c68-abce-c347836c804a",
        "guid": "68003",
        "linkIdentifier": null,
        "idxtype": "TAXON",
        "name": "Acacia nilotica",
        "kingdom": null,
        "nomenclaturalCode": "ICBN",
        "scientificName": "Acacia nilotica",
        "scientificNameAuthorship": "(L.) Delile",
        "author": "(L.) Delile",
        "nameComplete": "Acacia nilotica (l.) Delile",
        "nameFormatted": "<span class=\"scientific-name rank-species\">Acacia nilotica (l.) Delile</span>",
        "taxonomicStatus": "accepted",
        "nomenclaturalStatus": null,
        "parentGuid": "67862",
        "rank": "species",
        "rankID": 7000,
        "commonName": "Prickly Acacia",
        "commonNameSingle": "Prickly Acacia",
        "occurrenceCount": null,
        "conservationStatus": null,
        "infoSourceName": "NAQS",
        "infoSourceURL": "https://collections.ala.org.au/public/show/naqs",
        "superclass": "Magnoliidra",
        "orderGuid": "81173",
        "order": "Fabales",
        "class": "Magnoliopsida",
        "genus": "Acacia",
        "subclass": "Rosidae",
        "genusGuid": "67862",
        "family": "Mimosaceae",
        "classGuid": "22",
        "familyGuid": "67802",
        "superclassGuid": "21",
        "subclassGuid": "102360",
        "species": "Acacia nilotica",
        "speciesGuid": "68003",
        "highlight": "<b>Acacia</b> <b>nilotica</b> (l.) Delile<br><b>Acacia</b> <b>nilotica</b><br><b>Acacia nilotica</b>"
      },
      {
        "id": "23682bfd-d0d6-44ab-9c45-361c4c7ecdf8",
        "guid": "68004",
        "linkIdentifier": "Acacia nilotica indica",
        "idxtype": "TAXON",
        "name": "Acacia nilotica indica",
        "kingdom": null,
        "nomenclaturalCode": "ICBN",
        "scientificName": "Acacia nilotica indica",
        "scientificNameAuthorship": "(L.) Delile",
        "author": "(L.) Delile",
        "nameComplete": "Acacia nilotica indica (l.) Delile",
        "nameFormatted": "<span class=\"scientific-name rank-subspecies\">Acacia nilotica indica (l.) Delile</span>",
        "taxonomicStatus": "accepted",
        "nomenclaturalStatus": null,
        "parentGuid": "68003",
        "rank": "subspecies",
        "rankID": 8000,
        "commonName": "",
        "commonNameSingle": "",
        "occurrenceCount": null,
        "conservationStatus": null,
        "infoSourceName": "NAQS",
        "infoSourceURL": "https://collections.ala.org.au/public/show/naqs",
        "superclass": "Magnoliidra",
        "orderGuid": "81173",
        "order": "Fabales",
        "class": "Magnoliopsida",
        "genus": "Acacia",
        "subclass": "Rosidae",
        "species": "Acacia nilotica",
        "genusGuid": "67862",
        "family": "Mimosaceae",
        "classGuid": "22",
        "familyGuid": "67802",
        "speciesGuid": "68003",
        "superclassGuid": "21",
        "subclassGuid": "102360",
        "subspecies": "Acacia nilotica indica",
        "subspeciesGuid": "68004",
        "highlight": "<b>Acacia nilotica indica</b><br><b>Acacia</b> <b>nilotica</b> indica (l.) Delile<br><b>Acacia</b> <b>nilotica</b> indica"
      },
      {
        "id": "20be100e-1242-4ba6-bbed-5ba175abf350",
        "guid": "20be100e-1242-4ba6-bbed-5ba175abf350",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Catmint",
        "description": null,
        "taxonGuid": "90627"
      },
      {
        "id": "134e9600-a4bf-46f0-93c4-100e242584ce",
        "guid": "134e9600-a4bf-46f0-93c4-100e242584ce",
        "linkIdentifier": "Basket Plant",
        "idxtype": "COMMON",
        "name": "Basket Plant",
        "description": null,
        "taxonGuid": "85819"
      },
      {
        "id": "b199a622-6c56-491d-85ff-a100f7365b33",
        "guid": "b199a622-6c56-491d-85ff-a100f7365b33",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Mango Bud Mite",
        "description": null,
        "taxonGuid": "52488"
      },
      {
        "id": "addd8c4c-b0b7-4faa-b982-d3a902a100b2",
        "guid": "addd8c4c-b0b7-4faa-b982-d3a902a100b2",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Mopane Worm",
        "description": null,
        "taxonGuid": "79108"
      },
      {
        "id": "f6f74202-c5ea-4d83-b100-4977915a1a08",
        "guid": "f6f74202-c5ea-4d83-b100-4977915a1a08",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Japanese White Wax Scale",
        "description": null,
        "taxonGuid": "57455"
      },
      {
        "id": "954dd100-bdb1-44a3-8fcb-fa1fa1adc376",
        "guid": "954dd100-bdb1-44a3-8fcb-fa1fa1adc376",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Litchi Stink Bug",
        "description": null,
        "taxonGuid": "58307"
      },
      {
        "id": "f20de100-3f31-4a75-ad62-bf2d2812ca80",
        "guid": "f20de100-3f31-4a75-ad62-bf2d2812ca80",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Tobacco Leafminer",
        "description": null,
        "taxonGuid": "58977"
      },
      {
        "id": "7fc100e0-b23c-4b9a-8a76-c3bd909821a4",
        "guid": "7fc100e0-b23c-4b9a-8a76-c3bd909821a4",
        "linkIdentifier": "Sago Fern",
        "idxtype": "COMMON",
        "name": "Sago Fern",
        "description": null,
        "taxonGuid": "83269"
      }
    ],
    "queryTitle": "\"Acacia nilotica\""
  }
}
mbohun commented 5 years ago

...and just now ran into another one: https://ag-bie.oztaxa.com/ws/search.json?q=%22Solenopsis%20geminata%22

{
  "searchResults": {
    "totalRecords": 19,
    "facetResults": [],
    "results": [
      {
        "id": "a7f385cb-fe0f-4ff0-8ebd-ec509418cbee",
        "guid": "58524",
        "linkIdentifier": null,
        "idxtype": "TAXON",
        "name": "Solenopsis geminata",
        "kingdom": null,
        "nomenclaturalCode": "ICZN",
        "scientificName": "Solenopsis geminata",
        "scientificNameAuthorship": "(Fabricius, 1804)",
        "author": "(Fabricius, 1804)",
        "nameComplete": "Solenopsis geminata (Fabricius, 1804)",
        "nameFormatted": "<span class=\"scientific-name rank-species\"><span class=\"name\">Solenopsis geminata</span> <span class=\"author\">(Fabricius, 1804)</span></span>",
        "taxonomicStatus": "accepted",
        "nomenclaturalStatus": null,
        "parentGuid": "58523",
        "rank": "species",
        "rankID": 7000,
        "commonName": "Little Fire Ant, Tropical Fire Ant",
        "commonNameSingle": "Little Fire Ant",
        "occurrenceCount": null,
        "conservationStatus": null,
        "infoSourceName": "NAQS",
        "infoSourceURL": "https://collections.ala.org.au/public/show/naqs",
        "subkingdom": "Metazoa",
        "subfamily": "Myrmicinae",
        "suborder": "Apocrita",
        "subkingdomGuid": "106786",
        "genus": "Solenopsis",
        "genusGuid": "58523",
        "classGuid": "12",
        "superfamilyGuid": "101201",
        "superclassGuid": "100975",
        "superclass": "Hexapoda",
        "orderGuid": "52196",
        "subfamilyGuid": "75523",
        "order": "Hymenoptera",
        "class": "Insecta",
        "phylum": "Arthropoda",
        "suborderGuid": "101196",
        "family": "Formicidae",
        "familyGuid": "52197",
        "phylumGuid": "6",
        "superfamily": "Vespoidea",
        "species": "Solenopsis geminata",
        "speciesGuid": "58524",
        "highlight": "<b>Solenopsis geminata</b><br><b>Solenopsis</b> <b>geminata</b><br><b>Solenopsis</b> <b>geminata</b> (Fabricius, 1804)"
      },
      {
        "id": "1a4777fe-87db-424c-9f77-15f8692232ff",
        "guid": "81331",
        "linkIdentifier": "Solenopsis geminata group",
        "idxtype": "TAXON",
        "name": "Solenopsis geminata group",
        "kingdom": null,
        "nomenclaturalCode": "ICZN",
        "scientificName": "Solenopsis geminata group",
        "scientificNameAuthorship": null,
        "author": null,
        "nameComplete": "Solenopsis geminata group",
        "nameFormatted": "<span class=\"scientific-name rank-species\">Solenopsis geminata group</span>",
        "taxonomicStatus": "accepted",
        "nomenclaturalStatus": null,
        "parentGuid": "58523",
        "rank": "species",
        "rankID": 7000,
        "commonName": "Tropical Fire Ants",
        "commonNameSingle": "Tropical Fire Ants",
        "occurrenceCount": null,
        "conservationStatus": null,
        "infoSourceName": "NAQS",
        "infoSourceURL": "https://collections.ala.org.au/public/show/naqs",
        "species": "Solenopsis geminata group",
        "speciesGuid": "81331",
        "highlight": "<b>Solenopsis</b> <b>geminata</b> group<br><b>Solenopsis geminata group</b>"
      },
      {
        "id": "11bbf019-5d17-420c-9866-a73fbad100a0",
        "guid": "11bbf019-5d17-420c-9866-a73fbad100a0",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Mites",
        "description": null,
        "taxonGuid": "13"
      },
      {
        "id": "2ad8900d-100d-43cf-bb16-433e73a82ec6",
        "guid": "2ad8900d-100d-43cf-bb16-433e73a82ec6",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Anthracnose",
        "description": null,
        "taxonGuid": "55819"
      },
      {
        "id": "5c358087-a50e-4658-b100-57b6e7092db0",
        "guid": "5c358087-a50e-4658-b100-57b6e7092db0",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Fat-Hen",
        "description": null,
        "taxonGuid": "70549"
      },
      {
        "id": "369bb100-424a-4276-923c-684d87a60896",
        "guid": "369bb100-424a-4276-923c-684d87a60896",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Flour Mite",
        "description": null,
        "taxonGuid": "64674"
      },
      {
        "id": "fb59671d-3c92-40ac-aff1-00d1e7042764",
        "guid": "fb59671d-3c92-40ac-aff1-00d1e7042764",
        "linkIdentifier": "Sweet Briar",
        "idxtype": "COMMON",
        "name": "Sweet Briar",
        "description": null,
        "taxonGuid": "68404"
      },
      {
        "id": "f635360d-89bc-4159-a730-7d7bb376f100",
        "guid": "f635360d-89bc-4159-a730-7d7bb376f100",
        "linkIdentifier": "Banana Leafroller",
        "idxtype": "COMMON",
        "name": "Banana Leafroller",
        "description": null,
        "taxonGuid": "59171"
      },
      {
        "id": "f1450afe-100f-4a7f-b402-b9b55f286f62",
        "guid": "f1450afe-100f-4a7f-b402-b9b55f286f62",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Oriental Scale",
        "description": null,
        "taxonGuid": "57626"
      },
      {
        "id": "a280b014-ea02-401a-9a10-0e1eae3813ca",
        "guid": "a280b014-ea02-401a-9a10-0e1eae3813ca",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Chestnut Blight",
        "description": null,
        "taxonGuid": "69744"
      }
    ],
    "queryTitle": "\"Solenopsis geminata\""
  }
}
mbohun commented 5 years ago

...and another one: https://ag-bie.oztaxa.com/ws/search.json?q=%22Theba%20pisana%22

{
  "searchResults": {
    "totalRecords": 18,
    "facetResults": [],
    "results": [
      {
        "id": "4897705a-dae8-4b17-bf81-0fbdd4dede21",
        "guid": "56826",
        "linkIdentifier": "Theba pisana",
        "idxtype": "TAXON",
        "name": "Theba pisana",
        "kingdom": null,
        "nomenclaturalCode": "ICZN",
        "scientificName": "Theba pisana",
        "scientificNameAuthorship": "(Müller, 1774)",
        "author": "(Müller, 1774)",
        "nameComplete": "Theba pisana (Müller, 1774)",
        "nameFormatted": "<span class=\"scientific-name rank-species\"><span class=\"name\">Theba pisana</span> <span class=\"author\">(M&uuml;ller, 1774)</span></span>",
        "taxonomicStatus": "accepted",
        "nomenclaturalStatus": null,
        "parentGuid": "56825",
        "rank": "species",
        "rankID": 7000,
        "commonName": "White Snail, Mediterranean Sandsnail, White Italian Snail, White Garden Snail, Sand Dune Snail",
        "commonNameSingle": "White Snail",
        "occurrenceCount": null,
        "conservationStatus": null,
        "infoSourceName": "NAQS",
        "infoSourceURL": "https://collections.ala.org.au/public/show/naqs",
        "suborder": "Stylommatophora",
        "infraorderGuid": "56750",
        "infraorder": "Sigmurethra",
        "genus": "Theba",
        "subclass": "Orthogastropoda",
        "genusGuid": "56825",
        "classGuid": "14",
        "superorder": "Heterobranchia",
        "superfamilyGuid": "101350",
        "superorderGuid": "101329",
        "orderGuid": "52795",
        "order": "Eupulmonata",
        "class": "Gastropoda",
        "suborderGuid": "101335",
        "family": "Helicidae",
        "familyGuid": "56798",
        "subclassGuid": "101326",
        "superfamily": "Helicoidea",
        "species": "Theba pisana",
        "speciesGuid": "56826",
        "highlight": "<b>Theba</b> <b>pisana</b><br><b>Theba pisana</b><br><b>Theba</b> <b>pisana</b> (Müller, 1774)"
      },
      {
        "id": "11bbf019-5d17-420c-9866-a73fbad100a0",
        "guid": "11bbf019-5d17-420c-9866-a73fbad100a0",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Mites",
        "description": null,
        "taxonGuid": "13"
      },
      {
        "id": "2ad8900d-100d-43cf-bb16-433e73a82ec6",
        "guid": "2ad8900d-100d-43cf-bb16-433e73a82ec6",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Anthracnose",
        "description": null,
        "taxonGuid": "55819"
      },
      {
        "id": "5c358087-a50e-4658-b100-57b6e7092db0",
        "guid": "5c358087-a50e-4658-b100-57b6e7092db0",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Fat-Hen",
        "description": null,
        "taxonGuid": "70549"
      },
      {
        "id": "369bb100-424a-4276-923c-684d87a60896",
        "guid": "369bb100-424a-4276-923c-684d87a60896",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Flour Mite",
        "description": null,
        "taxonGuid": "64674"
      },
      {
        "id": "fb59671d-3c92-40ac-aff1-00d1e7042764",
        "guid": "fb59671d-3c92-40ac-aff1-00d1e7042764",
        "linkIdentifier": "Sweet Briar",
        "idxtype": "COMMON",
        "name": "Sweet Briar",
        "description": null,
        "taxonGuid": "68404"
      },
      {
        "id": "f635360d-89bc-4159-a730-7d7bb376f100",
        "guid": "f635360d-89bc-4159-a730-7d7bb376f100",
        "linkIdentifier": "Banana Leafroller",
        "idxtype": "COMMON",
        "name": "Banana Leafroller",
        "description": null,
        "taxonGuid": "59171"
      },
      {
        "id": "f1450afe-100f-4a7f-b402-b9b55f286f62",
        "guid": "f1450afe-100f-4a7f-b402-b9b55f286f62",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Oriental Scale",
        "description": null,
        "taxonGuid": "57626"
      },
      {
        "id": "a280b014-ea02-401a-9a10-0e1eae3813ca",
        "guid": "a280b014-ea02-401a-9a10-0e1eae3813ca",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Chestnut Blight",
        "description": null,
        "taxonGuid": "69744"
      },
      {
        "id": "7f754dd5-6b1f-46be-a4f3-100d403b7080",
        "guid": "7f754dd5-6b1f-46be-a4f3-100d403b7080",
        "linkIdentifier": null,
        "idxtype": "COMMON",
        "name": "Paperbark",
        "description": null,
        "taxonGuid": "88266"
      }
    ],
    "queryTitle": "\"Theba pisana\""
  }
}
charvolant commented 5 years ago

Confirm source DwCA for Acacia nilotica has 68003 (species) -> 67862 (genus) -> 67802 (family) -> 81173 (order) -> 102360 -> 22 (class) -> 21 -> 101427 -> 102343 (phylum) -> 102341 -> 20 (kingdom, Plantae)

charvolant commented 5 years ago

The solr log has lots of org.apache.solr.common.SolrException: ERROR: [doc=0488e3a1-a1aa-40dc-844e-26fe029252db] unknown field 'provenance' errors.

This is because there isn't a provenance field in /data/solr/bie/config/schema.xml which will cause a batch update to fail.

Updated https://github.com/AtlasOfLivingAustralia/ala-install/blob/ag-bie-install/ansible/roles/solr/files/solr/bie/conf/schema.xml to ensure it's handled correctly.

mbohun commented 5 years ago

In fact this issue / BUG provides a reasonable explanation (haven't verified it yet) why there is the difference between NAQS tblBiota.csv and ag-bie in the total number of records per kingdom:

moziauddin commented 4 years ago

This is now resolved. v0.6