search

AtlasOfLivingAustralia / biocache-store

Occurrence processing, indexing and batch processing
Other
7 stars 24 forks source link

SyncLocTable fails when run in parallel if a concurrent process creates a new Cassandra cl field first #311

Closed ansell closed 5 years ago

ansell commented 5 years ago

SyncLocTable is designed to be run using a single instance of biocache-store. It can fail when run concurrently on multiple instances if new spatial layers are added to Cassandra by one instance and the other instance complains that the column could not be created.

aws-bstore-1b 2019-01-18 08:40:38,052 INFO : [SyncLocTable] - cl10905 is NOT present
Exception in thread "main" com.datastax.driver.core.exceptions.InvalidQueryException: Invalid column name cl10905 because it conflicts with an existing column
    at com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:49)
    at com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:35)
    at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:293)
    at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:58)
    at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:39)
    at au.org.ala.biocache.persistence.Cassandra3PersistenceManager.addFieldToEntity(Cassandra3PersistenceManager.scala:605)
    at au.org.ala.biocache.tool.SyncLocTable$$anonfun$sync$1.apply(SyncLocTable.scala:40)
    at au.org.ala.biocache.tool.SyncLocTable$$anonfun$sync$1.apply(SyncLocTable.scala:35)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
    at au.org.ala.biocache.tool.SyncLocTable.sync(SyncLocTable.scala:35)
    at au.org.ala.biocache.tool.Sampling.sampling(Sampling.scala:451)
    at au.org.ala.biocache.tool.Sampling$.main(Sampling.scala:137)
    at au.org.ala.biocache.cmd.CMD2$.main(CMD2.scala:133)
    at au.org.ala.biocache.cmd.CMD2.main(CMD2.scala)
Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Invalid column name cl10905 because it conflicts with an existing column

It failed today with the same cl10905 on two of the three failed instances, and failed with a different field, cl10909 on the other. One instance succeeded in sampling its 25% of the records

adam-collins commented 5 years ago

I had forgotten that SyncLocTable is no longer required. To reduce resampling time the loc table only uses the 2 columns 'el' and 'cl'. Individual columns in the loc table for each el and cl are no longer required.