Closed adam-collins closed 4 years ago
There is nothing inherently special about that 0
. I don't mind working around it by changing it to a longer description label before loading if that would avoid having to track down an issue in biocache-store.
This is also a problem for single character miscPropertiesColumn
values and queryAssertionColumn
values.
Fixing for miscPropertiesColumn
may make the index slightly larger. Estimating as <1% larger.
Fixing for queryAssertionColumn
should have no impact because it looks like it is not in use.
Fixing for cl
will add up to 15 new values for up to 6 existing cl layers, including the referenced fire layer.
My preference is to reindex with https://github.com/AtlasOfLivingAustralia/biocache-store/pull/373
An alternative is to edit the fire shapefile, do a full resample, then reindex.
I would also prefer merging and deploying the biocache store fix as it will also patch up the other cl
layers you refer to that have been silently broken so far. I will merge, release, deploy, and reindex it today.
If it has unintended effects on existing indexed fields, the automatic swap collection jenkins job should pick it up and fail to swap.
On the index size concern, we have a buffer on disk of about 200GB on each node, which is effectively about 70GB for each of the three copies, so we should not expect to have issues there from some new small fields/values.
Released in biocache-store-2.4.7, deployed, and running complete reindex now:
http://aws-scjenkins.ala:9193/job/Complete%20Indexing/job/MASTER%20-%20Complete%20Re-index/628/
Running the complete reindex crashed zookeeper and solr, this was the error on one node http://aws-scjenkins.ala:9193/job/Complete%20Indexing/job/Complete%20Re-index/1917/console
:
aws-bstore-4b 2020-04-16 18:05:54,898 INFO : [IndexRunner] - FINAL >>> cassandraTime(s)=1903, processingTime[8](s)=130919, solrTime[2](s)=23978, totalTime(s)=16322, index docs committed/in ram/ram MB=11842500/41250/477, mem free(Mb)=4321, mem total(Mb)=12288, queues (processing/lucene docs/commit batch) 487/0/1
aws-bstore-4b 2020-04-16 18:05:55,189 INFO : [IndexRunner] - Total indexing time for this thread 276.8357 minutes. Records indexed: 23898824
aws-bstore-4b 2020-04-16 18:11:44,034 INFO : [IndexLocalNode] - Indexing completed in 282.68463 minutes
aws-bstore-4b 2020-04-16 18:11:45,148 INFO : [IndexLocalNode] - Writing 133 new fields into updated schema: /data/solr/solr-create/biocache/conf/schema.xml
aws-bstore-4b 2020-04-16 18:11:45,315 INFO : [SolrIndexDAO] - Initialising the solr server aws-zoo-a1.ala:2181,aws-zoo-b1.ala:2181,aws-zoo-b2.ala:2181,aws-zoo-c1.ala:2181,aws-zoo-c2.ala:2181 cloudserver:null solrServer:null
aws-bstore-4b 2020-04-16 18:11:56,054 ERROR: [IndexLocalNode] - failed to add new fields into SOLR: aws-zoo-a1.ala:2181,aws-zoo-b1.ala:2181,aws-zoo-b2.ala:2181,aws-zoo-c1.ala:2181,aws-zoo-c2.ala:2181
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper aws-zoo-a1.ala:2181,aws-zoo-b1.ala:2181,aws-zoo-b2.ala:2181,aws-zoo-c1.ala:2181,aws-zoo-c2.ala:2181 within 10000 ms
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:183)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:117)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:107)
at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:226)
at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:131)
at org.apache.solr.client.solrj.impl.CloudSolrClient.connect(CloudSolrClient.java:631)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1084)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
at org.apache.solr.client.solrj.SolrClient.ping(SolrClient.java:926)
at au.org.ala.biocache.index.SolrIndexDAO.init(SolrIndexDAO.scala:130)
at au.org.ala.biocache.index.SolrIndexDAO.addFieldToSolr(SolrIndexDAO.scala:212)
at au.org.ala.biocache.index.IndexLocalNode$$anonfun$importAdditionalFieldsToSOLR$1.apply(IndexLocalNode.scala:210)
at au.org.ala.biocache.index.IndexLocalNode$$anonfun$importAdditionalFieldsToSOLR$1.apply(IndexLocalNode.scala:208)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at au.org.ala.biocache.index.IndexLocalNode.importAdditionalFieldsToSOLR(IndexLocalNode.scala:208)
at au.org.ala.biocache.index.IndexLocalNode.indexRecords(IndexLocalNode.scala:126)
at au.org.ala.biocache.tool.IndexLocalRecordsV2$.main(IndexLocalRecordsV2.scala:88)
at au.org.ala.biocache.cmd.CMD2$.main(CMD2.scala:130)
at au.org.ala.biocache.cmd.CMD2.main(CMD2.scala)
Caused by: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper aws-zoo-a1.ala:2181,aws-zoo-b1.ala:2181,aws-zoo-b2.ala:2181,aws-zoo-c1.ala:2181,aws-zoo-c2.ala:2181 within 10000 ms
at org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:233)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:175)
... 20 more
aws-bstore-4b 2020-04-16 18:11:56,059 INFO : [IndexLocalNode] - Indexing complete. Records indexed: 23898824
This is the start of the errors on another node, which didn't make it to the "Indexing complete." message like the other node did http://aws-scjenkins.ala:9193/job/Complete%20Indexing/job/Complete%20Re-index/1915/consoleFull
:
aws-bstore-2b 2020-04-16 17:33:56,280 INFO : [IndexRunner] - FINAL >>> cassandraTime(s)=1449, processingTime[8](s)=115388, solrTime[2](s)=21135, totalTime(s)=14402, index docs committed/in ram/ram MB=10266250/102500/1027, mem free(Mb)=3296, mem total(Mb)=12288, queues (processing/lucene docs/commit batch) 381/0/1
aws-bstore-2b 2020-04-16 17:34:13,881 INFO : [IndexRunner] - Total indexing time for this thread 245.13452 minutes. Records indexed: 20683431
aws-bstore-2b 2020-04-16 17:34:57,896 INFO : [IndexLocalNode] - Indexing completed in 245.90057 minutes
aws-bstore-2b 2020-04-16 17:34:58,667 INFO : [IndexLocalNode] - Writing 201 new fields into updated schema: /data/solr/solr-create/biocache/conf/schema.xml
aws-bstore-2b 2020-04-16 17:34:58,900 INFO : [SolrIndexDAO] - Initialising the solr server aws-zoo-a1.ala:2181,aws-zoo-b1.ala:2181,aws-zoo-b2.ala:2181,aws-zoo-c1.ala:2181,aws-zoo-c2.ala:2181 cloudserver:null solrServer:null
aws-bstore-2b 2020-04-16 17:34:59,031 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc3b.ala:8983/solr/biocache: No such path /schema/fields/_test_diary_27__number_infested, retry? 0
aws-bstore-2b 2020-04-16 17:34:59,033 INFO : [SolrIndexDAO] - Field not in schema: _test_diary_27__number_infested
aws-bstore-2b 2020-04-16 17:34:59,038 INFO : [SolrIndexDAO] - Adding field: _test_diary_27__number_infested
aws-bstore-2b 2020-04-16 17:35:03,605 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc1b.ala:8983/solr/biocache: No such path /schema/fields/_abundance, retry? 0
aws-bstore-2b 2020-04-16 17:35:03,606 INFO : [SolrIndexDAO] - Field not in schema: _abundance
aws-bstore-2b 2020-04-16 17:35:03,606 INFO : [SolrIndexDAO] - Adding field: _abundance
aws-bstore-2b 2020-04-16 17:35:06,096 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc3b.ala:8983/solr/biocache: No such path /schema/fields/_test_diary_9_step_number, retry? 0
Have put a complete shutdown on the jenkins queue until we can look into it again tomorrow. If the fix isn't simple, we can revert to the previous version until it is fixed, although it was previously using a snapshot, so it isn't easy to reliably do that other than manually switching symlinks.
This error, the 404, is normal behaviour when testing if a field is in the live SOLR index.
Unfortunately after adding 58 new fields into the SOLR index, SOLR stopped responding.
Batch field additions appear to resolve this issue. I am still working on this.
While not necessary to function, the Jenkins job Create Solr Schema and Collection
needs an update so that it uses the SOLR collection biocache conf
instead of bstore[0] conf
as the new collection schema. This is because each bstore
updates the SOLR collection biocache conf
during indexing and each bstore conf
may differ. This should fall back to bstore[0] conf
when the SOLR collection biocache conf
is unavailable, e.g. indexing on an empty SOLR instance.
The issue for documenting (and possibly fixing) the wide array of locations that the solr schema is managed in and is copied to is being tracked in #275
It needs work, but may be superceded by whatever the mechanism is for the infrastructure upgrade project solr indexing step which does live indexing rather than offline indexing+copy+zookeeper registration of the collection.
The crash while updating fields occurred again during the reindex this morning:
aws-bstore-2b 2020-05-14 07:46:10,480 INFO : [Cassandra3PersistenceManager] - All threads have completed paging
aws-bstore-2b 2020-05-14 07:46:10,508 INFO : [IndexRunner] - FINAL >>> cassandraTime(s)=1703, processingTime[8](s)=112736, solrTime[2](s)=20377, totalTime(s)=13892, index docs committed/in ram/ram MB=10258750/55000/615, mem free(Mb)=4593, mem total(Mb)=12288, queues (processing/lucene docs/commit batch) 333/0/94
aws-bstore-2b 2020-05-14 07:46:10,851 INFO : [IndexRunner] - Total indexing time for this thread 238.42458 minutes. Records indexed: 20711667
aws-bstore-2b 2020-05-14 07:47:17,555 INFO : [IndexLocalNode] - Indexing completed in 239.57076 minutes
aws-bstore-2b 2020-05-14 07:47:18,415 INFO : [IndexLocalNode] - Writing 380 new fields into updated schema: /data/solr/solr-create/biocache/conf/schema.xml
aws-bstore-2b 2020-05-14 07:47:18,587 INFO : [SolrIndexDAO] - Initialising the solr server aws-zoo-a1.ala:2181,aws-zoo-b1.ala:2181,aws-zoo-b2.ala:2181,aws-zoo-c1.ala:2181,aws-zoo-c2.ala:2181 cloudserver:null solrServer:null
aws-bstore-2b 2020-05-14 07:47:18,704 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc2b.ala:8983/solr/biocache: No such path /schema/fields/_test_diary_5_test_number_mouldy, retry? 0
Create Solr Schema and Collection
. This will work with or without the following pull request.
Indexing is failing for contextual layer values with a single character.
For example, the value "0" with layer https://github.com/AtlasOfLivingAustralia/layers-service/issues/116