search

AtlasOfLivingAustralia / biocache-store

Occurrence processing, indexing and batch processing
Other
7 stars 24 forks source link

Removal of solr conf directory tries to access biocache1 collection #329

Closed ansell closed 5 years ago

ansell commented 5 years ago

The advice given to get a solr schema to add new fields was to delete the conf directory. The command that was used to remove the directory was:

rm -rf /data/solr/solr-create/biocache/conf

Doing this causes an error that appears to include a hardcoded solr collection name (or one that is in a configuration file that I haven't looked at):

aws-bstore-1b 2019-03-16 06:25:26,128 INFO : [IndexLocalNode] - Indexing completed in 217.47856 minutes
aws-bstore-1b 2019-03-16 06:25:26,612 INFO : [IndexLocalNode] - Writing 228 new fields into updated schema: /data/solr/solr-create/biocache/conf/schema.xml
aws-bstore-1b 2019-03-16 06:25:26,802 INFO : [SolrIndexDAO] - Initialising the solr server aws-zoo-a1.ala:2181,aws-zoo-b1.ala:2181,aws-zoo-b2.ala:2181,aws-zoo-c1.ala:2181,aws-zoo-c2.ala:2181 cloudserver:null solrServer:null
aws-bstore-1b 2019-03-16 06:25:27,032 ERROR: [IndexLocalNode] - failed to add new fields into SOLR: aws-zoo-a1.ala:2181,aws-zoo-b1.ala:2181,aws-zoo-b2.ala:2181,aws-zoo-c1.ala:2181,aws-zoo-c2.ala:2181
org.apache.solr.common.SolrException: Collection not found: biocache1
    at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1401)
    at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1094)
    at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1073)
    at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:160)
    at org.apache.solr.client.solrj.SolrClient.ping(SolrClient.java:926)
    at au.org.ala.biocache.index.SolrIndexDAO.init(SolrIndexDAO.scala:130)
    at au.org.ala.biocache.index.SolrIndexDAO.addFieldToSolr(SolrIndexDAO.scala:212)
    at au.org.ala.biocache.index.IndexLocalNode$$anonfun$importAdditionalFieldsToSOLR$1.apply(IndexLocalNode.scala:206)
    at au.org.ala.biocache.index.IndexLocalNode$$anonfun$importAdditionalFieldsToSOLR$1.apply(IndexLocalNode.scala:204)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at au.org.ala.biocache.index.IndexLocalNode.importAdditionalFieldsToSOLR(IndexLocalNode.scala:204)
    at au.org.ala.biocache.index.IndexLocalNode.indexRecords(IndexLocalNode.scala:123)
    at au.org.ala.biocache.tool.IndexLocalRecordsV2$.main(IndexLocalRecordsV2.scala:88)
    at au.org.ala.biocache.cmd.CMD2$.main(CMD2.scala:130)
    at au.org.ala.biocache.cmd.CMD2.main(CMD2.scala)
aws-bstore-1b 2019-03-16 06:25:27,036 INFO : [IndexLocalNode] - Indexing complete. Records indexed: 20968898
adam-collins commented 5 years ago

The correct conf directory to remove is /data/solr/biocache/conf https://github.com/AtlasOfLivingAustralia/biocache-store/blob/d74e8327b6b77ea6a49c7742e8d226d42c75b51a/src/main/scala/au/org/ala/biocache/tool/IndexLocalRecordsV2.scala#L32

These commits should fix this issue.

ansell commented 5 years ago

@adam-collins Travis had difficulty with this line:

https://travis-ci.org/AtlasOfLivingAustralia/biocache-store/jobs/508201746#L3227

adam-collins commented 5 years ago

Had my branches crossed. Better now.

ansell commented 5 years ago

Thanks for fixing the jenkins config. Deploying to aws-bstore instances and the usual wednesday morning complete reindex will run during the next 12 hours to test it out.

ansell commented 5 years ago

The Complete Reindex job that ran this morning using the latest snapshot from the master branch (https://github.com/AtlasOfLivingAustralia/biocache-store/commit/0d81b34e0baae700285a421df4fa9c301dd0fa10) failed just after it started with the following exception. The following is the full log, incase there was something else that was unexpected in the way the jenkins job is setup:

+ echo 'Removing old directories first.'
Removing old directories first.
+ rm -rf '/data/solr/merged_*'
+ rm -rf /data/solr/solr-create/biocache/data0- /data/solr/solr-create/biocache/data1-
+ rm -rf /data/solr/biocache/conf
+ echo 'Indexing started.'
Indexing started.
+ biocache index-local-node -t 8 -pt 8 -wc 2 -wt 2 -r 1024 -ps 500 -pb 500 -wb 500 -ws 100000000 -ms 0 -max -1
aws-bstore-1b 2019-03-20 02:47:55,980 INFO : [ConfigModule] - Using config file: /data/biocache/config/biocache-config.properties
aws-bstore-1b 2019-03-20 02:47:57,408 INFO : [Config] - Using the default set of blacklisted media URLs
aws-bstore-1b 2019-03-20 02:47:58,470 INFO : [SolrIndexDAO] - Initialising the solr server aws-zoo-a1.ala:2181,aws-zoo-b1.ala:2181,aws-zoo-b2.ala:2181,aws-zoo-c1.ala:2181,aws-zoo-c2.ala:2181 cloudserver:null solrServer:null
aws-bstore-1b 2019-03-20 02:47:59,511 INFO : [Config] - Number of fields to sample: 487
aws-bstore-1b 2019-03-20 02:47:59,512 INFO : [Config] - Fields to sample: cl3,el663,cl2125,cl2124,el603,cl938,el713,el1004,el1003,cl962,el2126,el2122,el2121,el2119,el898,el899,el1055,el1056,el2044,el2043,el2042,el2104,el1002,el2103,el2102,el2101,el2100,el2099,el2098,el2097,el2096,el2095,el2094,el2093,el2092,el2091,el2090,el2089,el1072,el2088,el2018,el2016,el2017,el1081,el1080,el1006,cl10907,el1079,el1078,el1077,el1074,el1073,el1047,el1046,el1045,el1044,el1043,el1042,el1041,el1040,el1039,el1038,el1037,el1036,el1035,el1034,el1033,el1032,el1031,el1030,el1029,el1028,el1027,el1026,el1025,el1024,el1023,el1022,el1021,el1020,el1019,el1018,el1017,el1016,el1015,el1014,el1005,el1013,el1012,el1011,el1010,el1009,el1008,el1007,el1001,el1000,el999,el998,el997,el996,el995,el994,el993,el992,el989,cl960,el615,el609,el665,el655,el653,el642,el645,el652,el651,el646,el644,el648,el643,el649,el654,el650,el656,el957,el955,el836,el954,el818,el815,el826,el812,el816,el844,el839,el827,el828,el811,el817,el841,el824,el843,el831,el822,el821,el840,el834,el842,el833,el825,el832,el835,el808,el809,el807,el838,el823,el830,el820,el819,el712,el813,el810,el837,el845,el829,el814,el741,el806,el952,el951,el950,el949,el948,el659,el947,el608,el586,el607,el610,el616,el658,el660,el705,el704,el683,el740,el669,el681,el667,el675,el672,el595,el599,el600,el593,el601,el596,el594,el602,el598,el662,el676,el591,el661,el682,el673,el668,el597,el671,el674,el666,el670,el647,el680,el723,cl10905,cl961,el717,el710,el799,el784,el787,el781,el789,el783,el790,el785,el786,el797,el782,el796,el794,el791,el798,el788,el795,el792,el793,el855,el850,el851,el852,el732,el849,el858,el853,el859,el856,el854,el860,el857,el727,el706,el714,el724,el738,el733,el757,el735,el739,el761,el762,el725,el776,el752,el760,el718,el766,el750,el778,el745,el772,el777,el709,el773,el758,el764,el734,el753,el756,el716,el759,el754,el779,el742,el746,el744,el749,el728,el743,el711,el726,el768,el775,el770,el748,el721,el708,el707,el737,el730,el731,el722,el729,el774,el769,el771,el767,el747,el720,el875,el755,el736,el751,el870,el862,el867,el879,el892,el873,el868,el864,el865,el885,el895,el883,el884,el891,el869,el894,el861,el877,el887,el871,el880,el881,el888,el863,cl10872,el878,el889,el886,el882,el872,el866,el893,el876,cl2109,el890,el874,el848,el765,el715,el719,el763,cl10847,cl10848,cl10874,cl10833,cl10835,cl10834,cl2022,cl10829,cl10831,cl10832,cl959,cl3004,cl3003,cl10910,cl2120,cl2111,cl2115,cl2116,cl2117,cl2113,cl2105,cl2110,cl2086,cl2087,cl2077,cl2083,cl2084,cl2085,cl2021,cl2078,cl2079,cl2076,cl2080,cl2081,cl10909,cl2020,cl2013,cl2050,cl2049,cl2048,cl10908,cl2045,cl1062,cl963,cl2015,cl2010,cl2009,cl2012,cl1085,cl1084,cl1050,cl1050b,cl1054,cl1071,cl1059,cl1058,cl987,cl1057,cl1070,cl1069,cl1068,cl1067,cl1066,cl1065,cl1064,cl1063,cl1053,cl1052,cl1049,cl1048,cl1051,cl1051b,cl605,cl611,cl612,cl604,cl613,cl614,cl606,cl917,cl618,cl678,cl988,cl942,cl941,cl937,cl940,cl936,cl958,cl991,cl990,cl964,cl965,cl10906,cl966,cl939,cl916,cl914,cl901,cl935,cl923,cl925,cl900,cl932,cl930,cl619,cl620,cl664,cl617,cl21,cl22,cl23,cl926,cl927,cl928,cl929,cl10900,cl10902,cl10903,cl1076a,cl1918,cl2052,cl20,cl1942,cl12021,cl10912,cl12078,cl12115,cl12079,cl12081,cl11061,cl12080
aws-bstore-1b 2019-03-20 02:47:59,732 INFO : [Config] - Number of fields to sample: 487
aws-bstore-1b 2019-03-20 02:47:59,732 INFO : [Config] - Fields to sample: cl3,el663,cl2125,cl2124,el603,cl938,el713,el1004,el1003,cl962,el2126,el2122,el2121,el2119,el898,el899,el1055,el1056,el2044,el2043,el2042,el2104,el1002,el2103,el2102,el2101,el2100,el2099,el2098,el2097,el2096,el2095,el2094,el2093,el2092,el2091,el2090,el2089,el1072,el2088,el2018,el2016,el2017,el1081,el1080,el1006,cl10907,el1079,el1078,el1077,el1074,el1073,el1047,el1046,el1045,el1044,el1043,el1042,el1041,el1040,el1039,el1038,el1037,el1036,el1035,el1034,el1033,el1032,el1031,el1030,el1029,el1028,el1027,el1026,el1025,el1024,el1023,el1022,el1021,el1020,el1019,el1018,el1017,el1016,el1015,el1014,el1005,el1013,el1012,el1011,el1010,el1009,el1008,el1007,el1001,el1000,el999,el998,el997,el996,el995,el994,el993,el992,el989,cl960,el615,el609,el665,el655,el653,el642,el645,el652,el651,el646,el644,el648,el643,el649,el654,el650,el656,el957,el955,el836,el954,el818,el815,el826,el812,el816,el844,el839,el827,el828,el811,el817,el841,el824,el843,el831,el822,el821,el840,el834,el842,el833,el825,el832,el835,el808,el809,el807,el838,el823,el830,el820,el819,el712,el813,el810,el837,el845,el829,el814,el741,el806,el952,el951,el950,el949,el948,el659,el947,el608,el586,el607,el610,el616,el658,el660,el705,el704,el683,el740,el669,el681,el667,el675,el672,el595,el599,el600,el593,el601,el596,el594,el602,el598,el662,el676,el591,el661,el682,el673,el668,el597,el671,el674,el666,el670,el647,el680,el723,cl10905,cl961,el717,el710,el799,el784,el787,el781,el789,el783,el790,el785,el786,el797,el782,el796,el794,el791,el798,el788,el795,el792,el793,el855,el850,el851,el852,el732,el849,el858,el853,el859,el856,el854,el860,el857,el727,el706,el714,el724,el738,el733,el757,el735,el739,el761,el762,el725,el776,el752,el760,el718,el766,el750,el778,el745,el772,el777,el709,el773,el758,el764,el734,el753,el756,el716,el759,el754,el779,el742,el746,el744,el749,el728,el743,el711,el726,el768,el775,el770,el748,el721,el708,el707,el737,el730,el731,el722,el729,el774,el769,el771,el767,el747,el720,el875,el755,el736,el751,el870,el862,el867,el879,el892,el873,el868,el864,el865,el885,el895,el883,el884,el891,el869,el894,el861,el877,el887,el871,el880,el881,el888,el863,cl10872,el878,el889,el886,el882,el872,el866,el893,el876,cl2109,el890,el874,el848,el765,el715,el719,el763,cl10847,cl10848,cl10874,cl10833,cl10835,cl10834,cl2022,cl10829,cl10831,cl10832,cl959,cl3004,cl3003,cl10910,cl2120,cl2111,cl2115,cl2116,cl2117,cl2113,cl2105,cl2110,cl2086,cl2087,cl2077,cl2083,cl2084,cl2085,cl2021,cl2078,cl2079,cl2076,cl2080,cl2081,cl10909,cl2020,cl2013,cl2050,cl2049,cl2048,cl10908,cl2045,cl1062,cl963,cl2015,cl2010,cl2009,cl2012,cl1085,cl1084,cl1050,cl1050b,cl1054,cl1071,cl1059,cl1058,cl987,cl1057,cl1070,cl1069,cl1068,cl1067,cl1066,cl1065,cl1064,cl1063,cl1053,cl1052,cl1049,cl1048,cl1051,cl1051b,cl605,cl611,cl612,cl604,cl613,cl614,cl606,cl917,cl618,cl678,cl988,cl942,cl941,cl937,cl940,cl936,cl958,cl991,cl990,cl964,cl965,cl10906,cl966,cl939,cl916,cl914,cl901,cl935,cl923,cl925,cl900,cl932,cl930,cl619,cl620,cl664,cl617,cl21,cl22,cl23,cl926,cl927,cl928,cl929,cl10900,cl10902,cl10903,cl1076a,cl1918,cl2052,cl20,cl1942,cl12021,cl10912,cl12078,cl12115,cl12079,cl12081,cl11061,cl12080
aws-bstore-1b 2019-03-20 02:47:59,784 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc3b.ala:8983/solr/biocache: No such path /schema/fields/cl3, retry? 0
aws-bstore-1b 2019-03-20 02:47:59,785 INFO : [SolrIndexDAO] - Field not in schema: cl3
aws-bstore-1b 2019-03-20 02:47:59,795 INFO : [SolrIndexDAO] - Adding field: cl3
aws-bstore-1b 2019-03-20 02:48:05,512 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc8b.ala:8983/solr/biocache: No such path /schema/fields/cl1942, retry? 0
aws-bstore-1b 2019-03-20 02:48:05,512 INFO : [SolrIndexDAO] - Field not in schema: cl1942
aws-bstore-1b 2019-03-20 02:48:05,512 INFO : [SolrIndexDAO] - Adding field: cl1942
aws-bstore-1b 2019-03-20 02:48:07,006 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc1b.ala:8983/solr/biocache: No such path /schema/fields/cl12078, retry? 0
aws-bstore-1b 2019-03-20 02:48:07,006 INFO : [SolrIndexDAO] - Field not in schema: cl12078
aws-bstore-1b 2019-03-20 02:48:07,006 INFO : [SolrIndexDAO] - Adding field: cl12078
aws-bstore-1b 2019-03-20 02:48:13,998 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc6b.ala:8983/solr/biocache: No such path /schema/fields/cl12081, retry? 0
aws-bstore-1b 2019-03-20 02:48:13,998 INFO : [SolrIndexDAO] - Field not in schema: cl12081
aws-bstore-1b 2019-03-20 02:48:13,998 INFO : [SolrIndexDAO] - Adding field: cl12081
aws-bstore-1b 2019-03-20 02:48:14,446 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc7b.ala:8983/solr/biocache: No such path /schema/fields/cl11061, retry? 0
aws-bstore-1b 2019-03-20 02:48:14,446 INFO : [SolrIndexDAO] - Field not in schema: cl11061
aws-bstore-1b 2019-03-20 02:48:14,446 INFO : [SolrIndexDAO] - Adding field: cl11061
Exception in thread "main" java.io.IOException: Error downloading files from zookeeper path /configs/biocache-2019-03-16-06-44 to /data/solr/biocache/conf
    at org.apache.solr.common.cloud.ZkMaintenanceUtils.downloadFromZK(ZkMaintenanceUtils.java:338)
    at org.apache.solr.common.cloud.SolrZkClient.downloadFromZK(SolrZkClient.java:828)
    at org.apache.solr.common.cloud.ZkConfigManager.downloadConfigDir(ZkConfigManager.java:89)
    at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.downloadConfig(ZkClientClusterStateProvider.java:105)
    at au.org.ala.biocache.index.IndexLocalNode.setUpSolrConfig(IndexLocalNode.scala:299)
    at au.org.ala.biocache.index.IndexLocalNode.indexRecords(IndexLocalNode.scala:74)
    at au.org.ala.biocache.tool.IndexLocalRecordsV2$.main(IndexLocalRecordsV2.scala:88)
    at au.org.ala.biocache.cmd.CMD2$.main(CMD2.scala:130)
    at au.org.ala.biocache.cmd.CMD2.main(CMD2.scala)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /configs/biocache-2019-03-16-06-44
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
    at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340)
    at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337)
    at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
    at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337)
    at org.apache.solr.common.cloud.ZkMaintenanceUtils.downloadFromZK(ZkMaintenanceUtils.java:310)
    ... 8 more
ansell commented 5 years ago

@M-Nicholls @adam-collins Reverting to biocache-store:2.4.4 and removing the rm -rf /data/solr/biocache/conf command failed to get an index generated. We are not able to run complete reindexing until we get a schema and solrconfig directory structure back again.

+ echo 'Removing old directories first.'
Removing old directories first.
+ rm -rf '/data/solr/merged_*'
+ rm -rf '/data/solr/solr-create/biocache/data*'
+ echo 'Indexing started.'
Indexing started.
+ biocache index-local-node -t 8 -pt 8 -wc 2 -wt 2 -r 1024 -ps 500 -pb 500 -wb 500 -ws 100000000 -ms 0 -max -1
aws-bstore-4b 2019-03-23 08:20:39,069 INFO : [ConfigModule] - Using config file: /data/biocache/config/biocache-config.properties
aws-bstore-4b 2019-03-23 08:20:40,066 INFO : [Config] - Using the default set of blacklisted media URLs
Exception in thread "main" org.apache.solr.common.SolrException: Error loading solr config from /data/solr/solr-create/biocache/conf/solrconfig.xml
    at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:188)
    at au.org.ala.biocache.index.IndexLocalNode.setUpSolrConfig(IndexLocalNode.scala:320)
    at au.org.ala.biocache.index.IndexLocalNode.indexRecords(IndexLocalNode.scala:73)
    at au.org.ala.biocache.tool.IndexLocalRecordsV2$.main(IndexLocalRecordsV2.scala:88)
    at au.org.ala.biocache.cmd.CMD2$.main(CMD2.scala:130)
    at au.org.ala.biocache.cmd.CMD2.main(CMD2.scala)
Caused by: org.apache.solr.core.SolrResourceNotFoundException: Can't find resource 'solrconfig.xml' in classpath or '/data/solr/solr-create/biocache'
    at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:405)
    at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:360)
    at org.apache.solr.core.Config.<init>(Config.java:120)
    at org.apache.solr.core.Config.<init>(Config.java:90)
    at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:203)
    at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:180)
    ... 5 more
ansell commented 5 years ago

Using the latest snapshot generated a new error, which I fixed by directly modifying the zookeeper entries for the current collection. The error was:

+ biocache index-local-node -t 8 -pt 8 -wc 2 -wt 2 -r 1024 -ps 500 -pb 500 -wb 500 -ws 100000000 -ms 0 -max -1
aws-bstore-4b 2019-03-26 10:41:28,376 INFO : [ConfigModule] - Using config file: /data/biocache/config/biocache-config.properties
aws-bstore-4b 2019-03-26 10:41:29,474 INFO : [Config] - Using the default set of blacklisted media URLs
aws-bstore-4b 2019-03-26 10:41:29,701 INFO : [SolrIndexDAO] - Initialising the solr server aws-zoo-a1.ala:2181,aws-zoo-b1.ala:2181,aws-zoo-b2.ala:2181,aws-zoo-c1.ala:2181,aws-zoo-c2.ala:2181 cloudserver:null solrServer:null
aws-bstore-4b 2019-03-26 10:41:30,573 INFO : [Config] - Number of fields to sample: 487
aws-bstore-4b 2019-03-26 10:41:30,574 INFO : [Config] - Fields to sample: cl3,el663,cl2125,cl2124,el603,cl938,el713,el1004,el1003,cl962,el2126,el2122,el2121,el2119,el898,el899,el1055,el1056,el2044,el2043,el2042,el2104,el1002,el2103,el2102,el2101,el2100,el2099,el2098,el2097,el2096,el2095,el2094,el2093,el2092,el2091,el2090,el2089,el1072,el2088,el2018,el2016,el2017,el1081,el1080,el1006,cl10907,el1079,el1078,el1077,el1074,el1073,el1047,el1046,el1045,el1044,el1043,el1042,el1041,el1040,el1039,el1038,el1037,el1036,el1035,el1034,el1033,el1032,el1031,el1030,el1029,el1028,el1027,el1026,el1025,el1024,el1023,el1022,el1021,el1020,el1019,el1018,el1017,el1016,el1015,el1014,el1005,el1013,el1012,el1011,el1010,el1009,el1008,el1007,el1001,el1000,el999,el998,el997,el996,el995,el994,el993,el992,el989,cl960,el615,el609,el665,el655,el653,el642,el645,el652,el651,el646,el644,el648,el643,el649,el654,el650,el656,el957,el955,el836,el954,el818,el815,el826,el812,el816,el844,el839,el827,el828,el811,el817,el841,el824,el843,el831,el822,el821,el840,el834,el842,el833,el825,el832,el835,el808,el809,el807,el838,el823,el830,el820,el819,el712,el813,el810,el837,el845,el829,el814,el741,el806,el952,el951,el950,el949,el948,el659,el947,el608,el586,el607,el610,el616,el658,el660,el705,el704,el683,el740,el669,el681,el667,el675,el672,el595,el599,el600,el593,el601,el596,el594,el602,el598,el662,el676,el591,el661,el682,el673,el668,el597,el671,el674,el666,el670,el647,el680,el723,cl10905,cl961,el717,el710,el799,el784,el787,el781,el789,el783,el790,el785,el786,el797,el782,el796,el794,el791,el798,el788,el795,el792,el793,el855,el850,el851,el852,el732,el849,el858,el853,el859,el856,el854,el860,el857,el727,el706,el714,el724,el738,el733,el757,el735,el739,el761,el762,el725,el776,el752,el760,el718,el766,el750,el778,el745,el772,el777,el709,el773,el758,el764,el734,el753,el756,el716,el759,el754,el779,el742,el746,el744,el749,el728,el743,el711,el726,el768,el775,el770,el748,el721,el708,el707,el737,el730,el731,el722,el729,el774,el769,el771,el767,el747,el720,el875,el755,el736,el751,el870,el862,el867,el879,el892,el873,el868,el864,el865,el885,el895,el883,el884,el891,el869,el894,el861,el877,el887,el871,el880,el881,el888,el863,cl10872,el878,el889,el886,el882,el872,el866,el893,el876,cl2109,el890,el874,el848,el765,el715,el719,el763,cl10847,cl10848,cl10874,cl10833,cl10835,cl10834,cl2022,cl10829,cl10831,cl10832,cl959,cl3004,cl3003,cl10910,cl2120,cl2111,cl2115,cl2116,cl2117,cl2113,cl2105,cl2110,cl2086,cl2087,cl2077,cl2083,cl2084,cl2085,cl2021,cl2078,cl2079,cl2076,cl2080,cl2081,cl10909,cl2020,cl2013,cl2050,cl2049,cl2048,cl10908,cl2045,cl1062,cl963,cl2015,cl2010,cl2009,cl2012,cl1085,cl1084,cl1050,cl1050b,cl1054,cl1071,cl1059,cl1058,cl987,cl1057,cl1070,cl1069,cl1068,cl1067,cl1066,cl1065,cl1064,cl1063,cl1053,cl1052,cl1049,cl1048,cl1051,cl1051b,cl605,cl611,cl612,cl604,cl613,cl614,cl606,cl917,cl618,cl678,cl988,cl942,cl941,cl937,cl940,cl936,cl958,cl991,cl990,cl964,cl965,cl10906,cl966,cl939,cl916,cl914,cl901,cl935,cl923,cl925,cl900,cl932,cl930,cl619,cl620,cl664,cl617,cl21,cl22,cl23,cl926,cl927,cl928,cl929,cl10900,cl10902,cl10903,cl1076a,cl1918,cl2052,cl20,cl1942,cl12021,cl10912,cl12078,cl12115,cl12079,cl12081,cl11061,cl12080
aws-bstore-4b 2019-03-26 10:41:30,694 INFO : [Config] - Number of fields to sample: 487
aws-bstore-4b 2019-03-26 10:41:30,694 INFO : [Config] - Fields to sample: cl3,el663,cl2125,cl2124,el603,cl938,el713,el1004,el1003,cl962,el2126,el2122,el2121,el2119,el898,el899,el1055,el1056,el2044,el2043,el2042,el2104,el1002,el2103,el2102,el2101,el2100,el2099,el2098,el2097,el2096,el2095,el2094,el2093,el2092,el2091,el2090,el2089,el1072,el2088,el2018,el2016,el2017,el1081,el1080,el1006,cl10907,el1079,el1078,el1077,el1074,el1073,el1047,el1046,el1045,el1044,el1043,el1042,el1041,el1040,el1039,el1038,el1037,el1036,el1035,el1034,el1033,el1032,el1031,el1030,el1029,el1028,el1027,el1026,el1025,el1024,el1023,el1022,el1021,el1020,el1019,el1018,el1017,el1016,el1015,el1014,el1005,el1013,el1012,el1011,el1010,el1009,el1008,el1007,el1001,el1000,el999,el998,el997,el996,el995,el994,el993,el992,el989,cl960,el615,el609,el665,el655,el653,el642,el645,el652,el651,el646,el644,el648,el643,el649,el654,el650,el656,el957,el955,el836,el954,el818,el815,el826,el812,el816,el844,el839,el827,el828,el811,el817,el841,el824,el843,el831,el822,el821,el840,el834,el842,el833,el825,el832,el835,el808,el809,el807,el838,el823,el830,el820,el819,el712,el813,el810,el837,el845,el829,el814,el741,el806,el952,el951,el950,el949,el948,el659,el947,el608,el586,el607,el610,el616,el658,el660,el705,el704,el683,el740,el669,el681,el667,el675,el672,el595,el599,el600,el593,el601,el596,el594,el602,el598,el662,el676,el591,el661,el682,el673,el668,el597,el671,el674,el666,el670,el647,el680,el723,cl10905,cl961,el717,el710,el799,el784,el787,el781,el789,el783,el790,el785,el786,el797,el782,el796,el794,el791,el798,el788,el795,el792,el793,el855,el850,el851,el852,el732,el849,el858,el853,el859,el856,el854,el860,el857,el727,el706,el714,el724,el738,el733,el757,el735,el739,el761,el762,el725,el776,el752,el760,el718,el766,el750,el778,el745,el772,el777,el709,el773,el758,el764,el734,el753,el756,el716,el759,el754,el779,el742,el746,el744,el749,el728,el743,el711,el726,el768,el775,el770,el748,el721,el708,el707,el737,el730,el731,el722,el729,el774,el769,el771,el767,el747,el720,el875,el755,el736,el751,el870,el862,el867,el879,el892,el873,el868,el864,el865,el885,el895,el883,el884,el891,el869,el894,el861,el877,el887,el871,el880,el881,el888,el863,cl10872,el878,el889,el886,el882,el872,el866,el893,el876,cl2109,el890,el874,el848,el765,el715,el719,el763,cl10847,cl10848,cl10874,cl10833,cl10835,cl10834,cl2022,cl10829,cl10831,cl10832,cl959,cl3004,cl3003,cl10910,cl2120,cl2111,cl2115,cl2116,cl2117,cl2113,cl2105,cl2110,cl2086,cl2087,cl2077,cl2083,cl2084,cl2085,cl2021,cl2078,cl2079,cl2076,cl2080,cl2081,cl10909,cl2020,cl2013,cl2050,cl2049,cl2048,cl10908,cl2045,cl1062,cl963,cl2015,cl2010,cl2009,cl2012,cl1085,cl1084,cl1050,cl1050b,cl1054,cl1071,cl1059,cl1058,cl987,cl1057,cl1070,cl1069,cl1068,cl1067,cl1066,cl1065,cl1064,cl1063,cl1053,cl1052,cl1049,cl1048,cl1051,cl1051b,cl605,cl611,cl612,cl604,cl613,cl614,cl606,cl917,cl618,cl678,cl988,cl942,cl941,cl937,cl940,cl936,cl958,cl991,cl990,cl964,cl965,cl10906,cl966,cl939,cl916,cl914,cl901,cl935,cl923,cl925,cl900,cl932,cl930,cl619,cl620,cl664,cl617,cl21,cl22,cl23,cl926,cl927,cl928,cl929,cl10900,cl10902,cl10903,cl1076a,cl1918,cl2052,cl20,cl1942,cl12021,cl10912,cl12078,cl12115,cl12079,cl12081,cl11061,cl12080
Exception in thread "main" org.apache.solr.common.SolrException: Can't load schema /data/solr/solr-create/biocache/conf/schema.xml: java.io.IOException: Is a directory
    at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:608)
    at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:182)
    at org.apache.solr.schema.ManagedIndexSchema.<init>(ManagedIndexSchema.java:104)
    at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:173)
    at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:45)
    at org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:75)
    at au.org.ala.biocache.index.IndexLocalNode.setUpSolrConfig(IndexLocalNode.scala:341)
    at au.org.ala.biocache.index.IndexLocalNode.indexRecords(IndexLocalNode.scala:76)
    at au.org.ala.biocache.tool.IndexLocalRecordsV2$.main(IndexLocalRecordsV2.scala:88)
    at au.org.ala.biocache.cmd.CMD2$.main(CMD2.scala:130)
    at au.org.ala.biocache.cmd.CMD2.main(CMD2.scala)
Caused by: org.apache.solr.common.SolrException: java.io.IOException: Is a directory
    at org.apache.solr.schema.ManagedIndexSchema.informResourceLoaderAwareObjectsInChain(ManagedIndexSchema.java:1316)
    at org.apache.solr.schema.ManagedIndexSchema.informResourceLoaderAwareObjectsForFieldType(ManagedIndexSchema.java:1167)
    at org.apache.solr.schema.ManagedIndexSchema.postReadInform(ManagedIndexSchema.java:1152)
    at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:604)
    ... 10 more
Caused by: java.io.IOException: Is a directory
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    at sun.nio.ch.IOUtil.read(IOUtil.java:197)
    at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
    at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:161)
    at java.io.BufferedReader.readLine(BufferedReader.java:324)
    at java.io.BufferedReader.readLine(BufferedReader.java:389)
    at org.apache.lucene.analysis.WordlistLoader.getLines(WordlistLoader.java:217)
    at org.apache.lucene.analysis.util.AbstractAnalysisFactory.getLines(AbstractAnalysisFactory.java:254)
    at org.apache.lucene.analysis.util.AbstractAnalysisFactory.getWordSet(AbstractAnalysisFactory.java:243)
    at org.apache.lucene.analysis.core.StopFilterFactory.inform(StopFilterFactory.java:96)
    at org.apache.solr.schema.ManagedIndexSchema.informResourceLoaderAwareObjectsInChain(ManagedIndexSchema.java:1314)
    ... 13 more

Three of the entries in both of the conf directories are directories instead of files, protwords.txt, stopwords.txt and synonyms.txt.

Those files in zookeeper were 0-byte sized entries and that seems to have been why they were being created on disk as directories rather than files. I modified them for the current biocache collection (biocache-2019-03-06-44)in zookeeper to contain only the apache commented headers from the examples in biocache-store/src/test/resources/solr/biocache/conf/ and they are now files on disk and the index process has proceeded past where it was failing.

Closing this as fixed as it is now downloading the configuration files to disk correctly from zookeeper. Further comments will be in #315 or others as necessary.