AtlasOfLivingAustralia / biocache-store

Occurrence processing, indexing and batch processing
Other
7 stars 24 forks source link

Incorrect column case during NZVH load #300

Closed ansell closed 5 years ago

ansell commented 5 years ago

The following occurred during a load of the NZVH (dr2153). The original Darwin Core Archive has the correct casing, accessRights in the TSV file, occurrence.txt, and uses the full IRI in the meta.xml file, http://purl.org/dc/terms/accessRights.

aws-bstore-1b 2018-11-27 13:30:03,341 ERROR: [Cassandra3PersistenceManager] - Incorrect column case. requested: 'accessrights', actual: 'accessRights', query: INSERT INTO occ (rowkey,"collectionID","geodeticDatum","stateProvince","catalogNumber","institutionCode","collectionCode","defaultValuesUsed","taxonRank","modified","country","basisOfRecord","taxonomicallyKosher","decimalLongitude","genus","verbatimEventDate","scientificName","family","lastModifiedTime","occurrenceID","decimalLatitude","typeStatus","eventDate","accessrights","taxonID","recordedBy","firstLoaded","miscProperties","dataResourceUid","locationDetermined","geospatiallyKosher") VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
aws-bstore-1b 2018-11-27 13:30:03,342 ERROR: [Cassandra3PersistenceManager] - Undefined column name accessrights
com.datastax.driver.core.exceptions.InvalidQueryException: Undefined column name accessrights
    at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
    at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
    at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:104)
    at au.org.ala.biocache.persistence.Cassandra3PersistenceManager.getPreparedStmt(Cassandra3PersistenceManager.scala:89)
    at au.org.ala.biocache.persistence.Cassandra3PersistenceManager.createPutStatement(Cassandra3PersistenceManager.scala:483)
    at au.org.ala.biocache.persistence.Cassandra3PersistenceManager.put(Cassandra3PersistenceManager.scala:413)
    at au.org.ala.biocache.persistence.Cassandra3PersistenceManager$$anonfun$putBatch$1.apply(Cassandra3PersistenceManager.scala:401)
    at au.org.ala.biocache.persistence.Cassandra3PersistenceManager$$anonfun$putBatch$1.apply(Cassandra3PersistenceManager.scala:395)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174)
    at au.org.ala.biocache.persistence.Cassandra3PersistenceManager.putBatch(Cassandra3PersistenceManager.scala:395)
    at au.org.ala.biocache.dao.OccurrenceDAOImpl.addRawOccurrenceBatch(OccurrenceDAOImpl.scala:578)
    at au.org.ala.biocache.load.DwCALoader$$anonfun$loadArchive$1.apply(DwCALoader.scala:286)
    at au.org.ala.biocache.load.DwCALoader$$anonfun$loadArchive$1.apply(DwCALoader.scala:228)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at au.org.ala.biocache.load.DwCALoader.loadArchive(DwCALoader.scala:228)
    at au.org.ala.biocache.load.DwCALoader$$anonfun$load$1.apply(DwCALoader.scala:126)
    at au.org.ala.biocache.load.DwCALoader$$anonfun$load$1.apply(DwCALoader.scala:117)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at au.org.ala.biocache.load.DwCALoader.load(DwCALoader.scala:117)
    at au.org.ala.biocache.load.Loader.load(Loader.scala:209)
    at au.org.ala.biocache.load.Loader$$anonfun$main$4.apply(Loader.scala:96)
    at au.org.ala.biocache.load.Loader$$anonfun$main$4.apply(Loader.scala:96)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
    at au.org.ala.biocache.load.Loader$.main(Loader.scala:95)
    at au.org.ala.biocache.cmd.CMD2$.main(CMD2.scala:133)
    at au.org.ala.biocache.cmd.CMD2.main(CMD2.scala)
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Undefined column name accessrights
    at com.datastax.driver.core.Responses$Error.asException(Responses.java:147)
    at com.datastax.driver.core.SessionManager$4.apply(SessionManager.java:220)
    at com.datastax.driver.core.SessionManager$4.apply(SessionManager.java:196)
    at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:211)
    at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:200)
    at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:111)
    at com.google.common.util.concurrent.MoreExecutors$5$1.run(MoreExecutors.java:992)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
aws-bstore-1b 2018-11-27 13:30:03,344 ERROR: [Cassandra3PersistenceManager] - Problem persisting the following to occ - Undefined column name accessrights
com.datastax.driver.core.exceptions.InvalidQueryException: Undefined column name accessrights
    at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
    at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
    at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:104)
    at au.org.ala.biocache.persistence.Cassandra3PersistenceManager.getPreparedStmt(Cassandra3PersistenceManager.scala:89)
    at au.org.ala.biocache.persistence.Cassandra3PersistenceManager.createPutStatement(Cassandra3PersistenceManager.scala:483)
    at au.org.ala.biocache.persistence.Cassandra3PersistenceManager.put(Cassandra3PersistenceManager.scala:413)
    at au.org.ala.biocache.persistence.Cassandra3PersistenceManager$$anonfun$putBatch$1.apply(Cassandra3PersistenceManager.scala:401)
    at au.org.ala.biocache.persistence.Cassandra3PersistenceManager$$anonfun$putBatch$1.apply(Cassandra3PersistenceManager.scala:395)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:174)
    at au.org.ala.biocache.persistence.Cassandra3PersistenceManager.putBatch(Cassandra3PersistenceManager.scala:395)
    at au.org.ala.biocache.dao.OccurrenceDAOImpl.addRawOccurrenceBatch(OccurrenceDAOImpl.scala:578)
    at au.org.ala.biocache.load.DwCALoader$$anonfun$loadArchive$1.apply(DwCALoader.scala:286)
    at au.org.ala.biocache.load.DwCALoader$$anonfun$loadArchive$1.apply(DwCALoader.scala:228)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at au.org.ala.biocache.load.DwCALoader.loadArchive(DwCALoader.scala:228)
    at au.org.ala.biocache.load.DwCALoader$$anonfun$load$1.apply(DwCALoader.scala:126)
    at au.org.ala.biocache.load.DwCALoader$$anonfun$load$1.apply(DwCALoader.scala:117)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at au.org.ala.biocache.load.DwCALoader.load(DwCALoader.scala:117)
    at au.org.ala.biocache.load.Loader.load(Loader.scala:209)
    at au.org.ala.biocache.load.Loader$$anonfun$main$4.apply(Loader.scala:96)
    at au.org.ala.biocache.load.Loader$$anonfun$main$4.apply(Loader.scala:96)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
    at au.org.ala.biocache.load.Loader$.main(Loader.scala:95)
    at au.org.ala.biocache.cmd.CMD2$.main(CMD2.scala:133)
    at au.org.ala.biocache.cmd.CMD2.main(CMD2.scala)
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Undefined column name accessrights
    at com.datastax.driver.core.Responses$Error.asException(Responses.java:147)
    at com.datastax.driver.core.SessionManager$4.apply(SessionManager.java:220)
    at com.datastax.driver.core.SessionManager$4.apply(SessionManager.java:196)
    at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:211)
    at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:200)
    at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:111)
    at com.google.common.util.concurrent.MoreExecutors$5$1.run(MoreExecutors.java:992)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
ansell commented 5 years ago

Fixed by @sadeghim