Closed ansell closed 5 years ago
The correct conf directory to remove is /data/solr/biocache/conf
https://github.com/AtlasOfLivingAustralia/biocache-store/blob/d74e8327b6b77ea6a49c7742e8d226d42c75b51a/src/main/scala/au/org/ala/biocache/tool/IndexLocalRecordsV2.scala#L32
These commits should fix this issue.
@adam-collins Travis had difficulty with this line:
https://travis-ci.org/AtlasOfLivingAustralia/biocache-store/jobs/508201746#L3227
Had my branches crossed. Better now.
Thanks for fixing the jenkins config. Deploying to aws-bstore
instances and the usual wednesday morning complete reindex will run during the next 12 hours to test it out.
The Complete Reindex job that ran this morning using the latest snapshot from the master branch (https://github.com/AtlasOfLivingAustralia/biocache-store/commit/0d81b34e0baae700285a421df4fa9c301dd0fa10) failed just after it started with the following exception. The following is the full log, incase there was something else that was unexpected in the way the jenkins job is setup:
+ echo 'Removing old directories first.'
Removing old directories first.
+ rm -rf '/data/solr/merged_*'
+ rm -rf /data/solr/solr-create/biocache/data0- /data/solr/solr-create/biocache/data1-
+ rm -rf /data/solr/biocache/conf
+ echo 'Indexing started.'
Indexing started.
+ biocache index-local-node -t 8 -pt 8 -wc 2 -wt 2 -r 1024 -ps 500 -pb 500 -wb 500 -ws 100000000 -ms 0 -max -1
aws-bstore-1b 2019-03-20 02:47:55,980 INFO : [ConfigModule] - Using config file: /data/biocache/config/biocache-config.properties
aws-bstore-1b 2019-03-20 02:47:57,408 INFO : [Config] - Using the default set of blacklisted media URLs
aws-bstore-1b 2019-03-20 02:47:58,470 INFO : [SolrIndexDAO] - Initialising the solr server aws-zoo-a1.ala:2181,aws-zoo-b1.ala:2181,aws-zoo-b2.ala:2181,aws-zoo-c1.ala:2181,aws-zoo-c2.ala:2181 cloudserver:null solrServer:null
aws-bstore-1b 2019-03-20 02:47:59,511 INFO : [Config] - Number of fields to sample: 487
aws-bstore-1b 2019-03-20 02:47:59,512 INFO : [Config] - Fields to sample: cl3,el663,cl2125,cl2124,el603,cl938,el713,el1004,el1003,cl962,el2126,el2122,el2121,el2119,el898,el899,el1055,el1056,el2044,el2043,el2042,el2104,el1002,el2103,el2102,el2101,el2100,el2099,el2098,el2097,el2096,el2095,el2094,el2093,el2092,el2091,el2090,el2089,el1072,el2088,el2018,el2016,el2017,el1081,el1080,el1006,cl10907,el1079,el1078,el1077,el1074,el1073,el1047,el1046,el1045,el1044,el1043,el1042,el1041,el1040,el1039,el1038,el1037,el1036,el1035,el1034,el1033,el1032,el1031,el1030,el1029,el1028,el1027,el1026,el1025,el1024,el1023,el1022,el1021,el1020,el1019,el1018,el1017,el1016,el1015,el1014,el1005,el1013,el1012,el1011,el1010,el1009,el1008,el1007,el1001,el1000,el999,el998,el997,el996,el995,el994,el993,el992,el989,cl960,el615,el609,el665,el655,el653,el642,el645,el652,el651,el646,el644,el648,el643,el649,el654,el650,el656,el957,el955,el836,el954,el818,el815,el826,el812,el816,el844,el839,el827,el828,el811,el817,el841,el824,el843,el831,el822,el821,el840,el834,el842,el833,el825,el832,el835,el808,el809,el807,el838,el823,el830,el820,el819,el712,el813,el810,el837,el845,el829,el814,el741,el806,el952,el951,el950,el949,el948,el659,el947,el608,el586,el607,el610,el616,el658,el660,el705,el704,el683,el740,el669,el681,el667,el675,el672,el595,el599,el600,el593,el601,el596,el594,el602,el598,el662,el676,el591,el661,el682,el673,el668,el597,el671,el674,el666,el670,el647,el680,el723,cl10905,cl961,el717,el710,el799,el784,el787,el781,el789,el783,el790,el785,el786,el797,el782,el796,el794,el791,el798,el788,el795,el792,el793,el855,el850,el851,el852,el732,el849,el858,el853,el859,el856,el854,el860,el857,el727,el706,el714,el724,el738,el733,el757,el735,el739,el761,el762,el725,el776,el752,el760,el718,el766,el750,el778,el745,el772,el777,el709,el773,el758,el764,el734,el753,el756,el716,el759,el754,el779,el742,el746,el744,el749,el728,el743,el711,el726,el768,el775,el770,el748,el721,el708,el707,el737,el730,el731,el722,el729,el774,el769,el771,el767,el747,el720,el875,el755,el736,el751,el870,el862,el867,el879,el892,el873,el868,el864,el865,el885,el895,el883,el884,el891,el869,el894,el861,el877,el887,el871,el880,el881,el888,el863,cl10872,el878,el889,el886,el882,el872,el866,el893,el876,cl2109,el890,el874,el848,el765,el715,el719,el763,cl10847,cl10848,cl10874,cl10833,cl10835,cl10834,cl2022,cl10829,cl10831,cl10832,cl959,cl3004,cl3003,cl10910,cl2120,cl2111,cl2115,cl2116,cl2117,cl2113,cl2105,cl2110,cl2086,cl2087,cl2077,cl2083,cl2084,cl2085,cl2021,cl2078,cl2079,cl2076,cl2080,cl2081,cl10909,cl2020,cl2013,cl2050,cl2049,cl2048,cl10908,cl2045,cl1062,cl963,cl2015,cl2010,cl2009,cl2012,cl1085,cl1084,cl1050,cl1050b,cl1054,cl1071,cl1059,cl1058,cl987,cl1057,cl1070,cl1069,cl1068,cl1067,cl1066,cl1065,cl1064,cl1063,cl1053,cl1052,cl1049,cl1048,cl1051,cl1051b,cl605,cl611,cl612,cl604,cl613,cl614,cl606,cl917,cl618,cl678,cl988,cl942,cl941,cl937,cl940,cl936,cl958,cl991,cl990,cl964,cl965,cl10906,cl966,cl939,cl916,cl914,cl901,cl935,cl923,cl925,cl900,cl932,cl930,cl619,cl620,cl664,cl617,cl21,cl22,cl23,cl926,cl927,cl928,cl929,cl10900,cl10902,cl10903,cl1076a,cl1918,cl2052,cl20,cl1942,cl12021,cl10912,cl12078,cl12115,cl12079,cl12081,cl11061,cl12080
aws-bstore-1b 2019-03-20 02:47:59,732 INFO : [Config] - Number of fields to sample: 487
aws-bstore-1b 2019-03-20 02:47:59,732 INFO : [Config] - Fields to sample: cl3,el663,cl2125,cl2124,el603,cl938,el713,el1004,el1003,cl962,el2126,el2122,el2121,el2119,el898,el899,el1055,el1056,el2044,el2043,el2042,el2104,el1002,el2103,el2102,el2101,el2100,el2099,el2098,el2097,el2096,el2095,el2094,el2093,el2092,el2091,el2090,el2089,el1072,el2088,el2018,el2016,el2017,el1081,el1080,el1006,cl10907,el1079,el1078,el1077,el1074,el1073,el1047,el1046,el1045,el1044,el1043,el1042,el1041,el1040,el1039,el1038,el1037,el1036,el1035,el1034,el1033,el1032,el1031,el1030,el1029,el1028,el1027,el1026,el1025,el1024,el1023,el1022,el1021,el1020,el1019,el1018,el1017,el1016,el1015,el1014,el1005,el1013,el1012,el1011,el1010,el1009,el1008,el1007,el1001,el1000,el999,el998,el997,el996,el995,el994,el993,el992,el989,cl960,el615,el609,el665,el655,el653,el642,el645,el652,el651,el646,el644,el648,el643,el649,el654,el650,el656,el957,el955,el836,el954,el818,el815,el826,el812,el816,el844,el839,el827,el828,el811,el817,el841,el824,el843,el831,el822,el821,el840,el834,el842,el833,el825,el832,el835,el808,el809,el807,el838,el823,el830,el820,el819,el712,el813,el810,el837,el845,el829,el814,el741,el806,el952,el951,el950,el949,el948,el659,el947,el608,el586,el607,el610,el616,el658,el660,el705,el704,el683,el740,el669,el681,el667,el675,el672,el595,el599,el600,el593,el601,el596,el594,el602,el598,el662,el676,el591,el661,el682,el673,el668,el597,el671,el674,el666,el670,el647,el680,el723,cl10905,cl961,el717,el710,el799,el784,el787,el781,el789,el783,el790,el785,el786,el797,el782,el796,el794,el791,el798,el788,el795,el792,el793,el855,el850,el851,el852,el732,el849,el858,el853,el859,el856,el854,el860,el857,el727,el706,el714,el724,el738,el733,el757,el735,el739,el761,el762,el725,el776,el752,el760,el718,el766,el750,el778,el745,el772,el777,el709,el773,el758,el764,el734,el753,el756,el716,el759,el754,el779,el742,el746,el744,el749,el728,el743,el711,el726,el768,el775,el770,el748,el721,el708,el707,el737,el730,el731,el722,el729,el774,el769,el771,el767,el747,el720,el875,el755,el736,el751,el870,el862,el867,el879,el892,el873,el868,el864,el865,el885,el895,el883,el884,el891,el869,el894,el861,el877,el887,el871,el880,el881,el888,el863,cl10872,el878,el889,el886,el882,el872,el866,el893,el876,cl2109,el890,el874,el848,el765,el715,el719,el763,cl10847,cl10848,cl10874,cl10833,cl10835,cl10834,cl2022,cl10829,cl10831,cl10832,cl959,cl3004,cl3003,cl10910,cl2120,cl2111,cl2115,cl2116,cl2117,cl2113,cl2105,cl2110,cl2086,cl2087,cl2077,cl2083,cl2084,cl2085,cl2021,cl2078,cl2079,cl2076,cl2080,cl2081,cl10909,cl2020,cl2013,cl2050,cl2049,cl2048,cl10908,cl2045,cl1062,cl963,cl2015,cl2010,cl2009,cl2012,cl1085,cl1084,cl1050,cl1050b,cl1054,cl1071,cl1059,cl1058,cl987,cl1057,cl1070,cl1069,cl1068,cl1067,cl1066,cl1065,cl1064,cl1063,cl1053,cl1052,cl1049,cl1048,cl1051,cl1051b,cl605,cl611,cl612,cl604,cl613,cl614,cl606,cl917,cl618,cl678,cl988,cl942,cl941,cl937,cl940,cl936,cl958,cl991,cl990,cl964,cl965,cl10906,cl966,cl939,cl916,cl914,cl901,cl935,cl923,cl925,cl900,cl932,cl930,cl619,cl620,cl664,cl617,cl21,cl22,cl23,cl926,cl927,cl928,cl929,cl10900,cl10902,cl10903,cl1076a,cl1918,cl2052,cl20,cl1942,cl12021,cl10912,cl12078,cl12115,cl12079,cl12081,cl11061,cl12080
aws-bstore-1b 2019-03-20 02:47:59,784 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc3b.ala:8983/solr/biocache: No such path /schema/fields/cl3, retry? 0
aws-bstore-1b 2019-03-20 02:47:59,785 INFO : [SolrIndexDAO] - Field not in schema: cl3
aws-bstore-1b 2019-03-20 02:47:59,795 INFO : [SolrIndexDAO] - Adding field: cl3
aws-bstore-1b 2019-03-20 02:48:05,512 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc8b.ala:8983/solr/biocache: No such path /schema/fields/cl1942, retry? 0
aws-bstore-1b 2019-03-20 02:48:05,512 INFO : [SolrIndexDAO] - Field not in schema: cl1942
aws-bstore-1b 2019-03-20 02:48:05,512 INFO : [SolrIndexDAO] - Adding field: cl1942
aws-bstore-1b 2019-03-20 02:48:07,006 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc1b.ala:8983/solr/biocache: No such path /schema/fields/cl12078, retry? 0
aws-bstore-1b 2019-03-20 02:48:07,006 INFO : [SolrIndexDAO] - Field not in schema: cl12078
aws-bstore-1b 2019-03-20 02:48:07,006 INFO : [SolrIndexDAO] - Adding field: cl12078
aws-bstore-1b 2019-03-20 02:48:13,998 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc6b.ala:8983/solr/biocache: No such path /schema/fields/cl12081, retry? 0
aws-bstore-1b 2019-03-20 02:48:13,998 INFO : [SolrIndexDAO] - Field not in schema: cl12081
aws-bstore-1b 2019-03-20 02:48:13,998 INFO : [SolrIndexDAO] - Adding field: cl12081
aws-bstore-1b 2019-03-20 02:48:14,446 ERROR: [CloudSolrClient] - Request to collection biocache failed due to (404) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc7b.ala:8983/solr/biocache: No such path /schema/fields/cl11061, retry? 0
aws-bstore-1b 2019-03-20 02:48:14,446 INFO : [SolrIndexDAO] - Field not in schema: cl11061
aws-bstore-1b 2019-03-20 02:48:14,446 INFO : [SolrIndexDAO] - Adding field: cl11061
Exception in thread "main" java.io.IOException: Error downloading files from zookeeper path /configs/biocache-2019-03-16-06-44 to /data/solr/biocache/conf
at org.apache.solr.common.cloud.ZkMaintenanceUtils.downloadFromZK(ZkMaintenanceUtils.java:338)
at org.apache.solr.common.cloud.SolrZkClient.downloadFromZK(SolrZkClient.java:828)
at org.apache.solr.common.cloud.ZkConfigManager.downloadConfigDir(ZkConfigManager.java:89)
at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.downloadConfig(ZkClientClusterStateProvider.java:105)
at au.org.ala.biocache.index.IndexLocalNode.setUpSolrConfig(IndexLocalNode.scala:299)
at au.org.ala.biocache.index.IndexLocalNode.indexRecords(IndexLocalNode.scala:74)
at au.org.ala.biocache.tool.IndexLocalRecordsV2$.main(IndexLocalRecordsV2.scala:88)
at au.org.ala.biocache.cmd.CMD2$.main(CMD2.scala:130)
at au.org.ala.biocache.cmd.CMD2.main(CMD2.scala)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /configs/biocache-2019-03-16-06-44
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:340)
at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:337)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:337)
at org.apache.solr.common.cloud.ZkMaintenanceUtils.downloadFromZK(ZkMaintenanceUtils.java:310)
... 8 more
@M-Nicholls @adam-collins Reverting to biocache-store:2.4.4 and removing the rm -rf /data/solr/biocache/conf
command failed to get an index generated. We are not able to run complete reindexing until we get a schema and solrconfig directory structure back again.
+ echo 'Removing old directories first.'
Removing old directories first.
+ rm -rf '/data/solr/merged_*'
+ rm -rf '/data/solr/solr-create/biocache/data*'
+ echo 'Indexing started.'
Indexing started.
+ biocache index-local-node -t 8 -pt 8 -wc 2 -wt 2 -r 1024 -ps 500 -pb 500 -wb 500 -ws 100000000 -ms 0 -max -1
aws-bstore-4b 2019-03-23 08:20:39,069 INFO : [ConfigModule] - Using config file: /data/biocache/config/biocache-config.properties
aws-bstore-4b 2019-03-23 08:20:40,066 INFO : [Config] - Using the default set of blacklisted media URLs
Exception in thread "main" org.apache.solr.common.SolrException: Error loading solr config from /data/solr/solr-create/biocache/conf/solrconfig.xml
at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:188)
at au.org.ala.biocache.index.IndexLocalNode.setUpSolrConfig(IndexLocalNode.scala:320)
at au.org.ala.biocache.index.IndexLocalNode.indexRecords(IndexLocalNode.scala:73)
at au.org.ala.biocache.tool.IndexLocalRecordsV2$.main(IndexLocalRecordsV2.scala:88)
at au.org.ala.biocache.cmd.CMD2$.main(CMD2.scala:130)
at au.org.ala.biocache.cmd.CMD2.main(CMD2.scala)
Caused by: org.apache.solr.core.SolrResourceNotFoundException: Can't find resource 'solrconfig.xml' in classpath or '/data/solr/solr-create/biocache'
at org.apache.solr.core.SolrResourceLoader.openResource(SolrResourceLoader.java:405)
at org.apache.solr.core.SolrResourceLoader.openConfig(SolrResourceLoader.java:360)
at org.apache.solr.core.Config.<init>(Config.java:120)
at org.apache.solr.core.Config.<init>(Config.java:90)
at org.apache.solr.core.SolrConfig.<init>(SolrConfig.java:203)
at org.apache.solr.core.SolrConfig.readFromResourceLoader(SolrConfig.java:180)
... 5 more
Using the latest snapshot generated a new error, which I fixed by directly modifying the zookeeper entries for the current collection. The error was:
+ biocache index-local-node -t 8 -pt 8 -wc 2 -wt 2 -r 1024 -ps 500 -pb 500 -wb 500 -ws 100000000 -ms 0 -max -1
aws-bstore-4b 2019-03-26 10:41:28,376 INFO : [ConfigModule] - Using config file: /data/biocache/config/biocache-config.properties
aws-bstore-4b 2019-03-26 10:41:29,474 INFO : [Config] - Using the default set of blacklisted media URLs
aws-bstore-4b 2019-03-26 10:41:29,701 INFO : [SolrIndexDAO] - Initialising the solr server aws-zoo-a1.ala:2181,aws-zoo-b1.ala:2181,aws-zoo-b2.ala:2181,aws-zoo-c1.ala:2181,aws-zoo-c2.ala:2181 cloudserver:null solrServer:null
aws-bstore-4b 2019-03-26 10:41:30,573 INFO : [Config] - Number of fields to sample: 487
aws-bstore-4b 2019-03-26 10:41:30,574 INFO : [Config] - Fields to sample: cl3,el663,cl2125,cl2124,el603,cl938,el713,el1004,el1003,cl962,el2126,el2122,el2121,el2119,el898,el899,el1055,el1056,el2044,el2043,el2042,el2104,el1002,el2103,el2102,el2101,el2100,el2099,el2098,el2097,el2096,el2095,el2094,el2093,el2092,el2091,el2090,el2089,el1072,el2088,el2018,el2016,el2017,el1081,el1080,el1006,cl10907,el1079,el1078,el1077,el1074,el1073,el1047,el1046,el1045,el1044,el1043,el1042,el1041,el1040,el1039,el1038,el1037,el1036,el1035,el1034,el1033,el1032,el1031,el1030,el1029,el1028,el1027,el1026,el1025,el1024,el1023,el1022,el1021,el1020,el1019,el1018,el1017,el1016,el1015,el1014,el1005,el1013,el1012,el1011,el1010,el1009,el1008,el1007,el1001,el1000,el999,el998,el997,el996,el995,el994,el993,el992,el989,cl960,el615,el609,el665,el655,el653,el642,el645,el652,el651,el646,el644,el648,el643,el649,el654,el650,el656,el957,el955,el836,el954,el818,el815,el826,el812,el816,el844,el839,el827,el828,el811,el817,el841,el824,el843,el831,el822,el821,el840,el834,el842,el833,el825,el832,el835,el808,el809,el807,el838,el823,el830,el820,el819,el712,el813,el810,el837,el845,el829,el814,el741,el806,el952,el951,el950,el949,el948,el659,el947,el608,el586,el607,el610,el616,el658,el660,el705,el704,el683,el740,el669,el681,el667,el675,el672,el595,el599,el600,el593,el601,el596,el594,el602,el598,el662,el676,el591,el661,el682,el673,el668,el597,el671,el674,el666,el670,el647,el680,el723,cl10905,cl961,el717,el710,el799,el784,el787,el781,el789,el783,el790,el785,el786,el797,el782,el796,el794,el791,el798,el788,el795,el792,el793,el855,el850,el851,el852,el732,el849,el858,el853,el859,el856,el854,el860,el857,el727,el706,el714,el724,el738,el733,el757,el735,el739,el761,el762,el725,el776,el752,el760,el718,el766,el750,el778,el745,el772,el777,el709,el773,el758,el764,el734,el753,el756,el716,el759,el754,el779,el742,el746,el744,el749,el728,el743,el711,el726,el768,el775,el770,el748,el721,el708,el707,el737,el730,el731,el722,el729,el774,el769,el771,el767,el747,el720,el875,el755,el736,el751,el870,el862,el867,el879,el892,el873,el868,el864,el865,el885,el895,el883,el884,el891,el869,el894,el861,el877,el887,el871,el880,el881,el888,el863,cl10872,el878,el889,el886,el882,el872,el866,el893,el876,cl2109,el890,el874,el848,el765,el715,el719,el763,cl10847,cl10848,cl10874,cl10833,cl10835,cl10834,cl2022,cl10829,cl10831,cl10832,cl959,cl3004,cl3003,cl10910,cl2120,cl2111,cl2115,cl2116,cl2117,cl2113,cl2105,cl2110,cl2086,cl2087,cl2077,cl2083,cl2084,cl2085,cl2021,cl2078,cl2079,cl2076,cl2080,cl2081,cl10909,cl2020,cl2013,cl2050,cl2049,cl2048,cl10908,cl2045,cl1062,cl963,cl2015,cl2010,cl2009,cl2012,cl1085,cl1084,cl1050,cl1050b,cl1054,cl1071,cl1059,cl1058,cl987,cl1057,cl1070,cl1069,cl1068,cl1067,cl1066,cl1065,cl1064,cl1063,cl1053,cl1052,cl1049,cl1048,cl1051,cl1051b,cl605,cl611,cl612,cl604,cl613,cl614,cl606,cl917,cl618,cl678,cl988,cl942,cl941,cl937,cl940,cl936,cl958,cl991,cl990,cl964,cl965,cl10906,cl966,cl939,cl916,cl914,cl901,cl935,cl923,cl925,cl900,cl932,cl930,cl619,cl620,cl664,cl617,cl21,cl22,cl23,cl926,cl927,cl928,cl929,cl10900,cl10902,cl10903,cl1076a,cl1918,cl2052,cl20,cl1942,cl12021,cl10912,cl12078,cl12115,cl12079,cl12081,cl11061,cl12080
aws-bstore-4b 2019-03-26 10:41:30,694 INFO : [Config] - Number of fields to sample: 487
aws-bstore-4b 2019-03-26 10:41:30,694 INFO : [Config] - Fields to sample: cl3,el663,cl2125,cl2124,el603,cl938,el713,el1004,el1003,cl962,el2126,el2122,el2121,el2119,el898,el899,el1055,el1056,el2044,el2043,el2042,el2104,el1002,el2103,el2102,el2101,el2100,el2099,el2098,el2097,el2096,el2095,el2094,el2093,el2092,el2091,el2090,el2089,el1072,el2088,el2018,el2016,el2017,el1081,el1080,el1006,cl10907,el1079,el1078,el1077,el1074,el1073,el1047,el1046,el1045,el1044,el1043,el1042,el1041,el1040,el1039,el1038,el1037,el1036,el1035,el1034,el1033,el1032,el1031,el1030,el1029,el1028,el1027,el1026,el1025,el1024,el1023,el1022,el1021,el1020,el1019,el1018,el1017,el1016,el1015,el1014,el1005,el1013,el1012,el1011,el1010,el1009,el1008,el1007,el1001,el1000,el999,el998,el997,el996,el995,el994,el993,el992,el989,cl960,el615,el609,el665,el655,el653,el642,el645,el652,el651,el646,el644,el648,el643,el649,el654,el650,el656,el957,el955,el836,el954,el818,el815,el826,el812,el816,el844,el839,el827,el828,el811,el817,el841,el824,el843,el831,el822,el821,el840,el834,el842,el833,el825,el832,el835,el808,el809,el807,el838,el823,el830,el820,el819,el712,el813,el810,el837,el845,el829,el814,el741,el806,el952,el951,el950,el949,el948,el659,el947,el608,el586,el607,el610,el616,el658,el660,el705,el704,el683,el740,el669,el681,el667,el675,el672,el595,el599,el600,el593,el601,el596,el594,el602,el598,el662,el676,el591,el661,el682,el673,el668,el597,el671,el674,el666,el670,el647,el680,el723,cl10905,cl961,el717,el710,el799,el784,el787,el781,el789,el783,el790,el785,el786,el797,el782,el796,el794,el791,el798,el788,el795,el792,el793,el855,el850,el851,el852,el732,el849,el858,el853,el859,el856,el854,el860,el857,el727,el706,el714,el724,el738,el733,el757,el735,el739,el761,el762,el725,el776,el752,el760,el718,el766,el750,el778,el745,el772,el777,el709,el773,el758,el764,el734,el753,el756,el716,el759,el754,el779,el742,el746,el744,el749,el728,el743,el711,el726,el768,el775,el770,el748,el721,el708,el707,el737,el730,el731,el722,el729,el774,el769,el771,el767,el747,el720,el875,el755,el736,el751,el870,el862,el867,el879,el892,el873,el868,el864,el865,el885,el895,el883,el884,el891,el869,el894,el861,el877,el887,el871,el880,el881,el888,el863,cl10872,el878,el889,el886,el882,el872,el866,el893,el876,cl2109,el890,el874,el848,el765,el715,el719,el763,cl10847,cl10848,cl10874,cl10833,cl10835,cl10834,cl2022,cl10829,cl10831,cl10832,cl959,cl3004,cl3003,cl10910,cl2120,cl2111,cl2115,cl2116,cl2117,cl2113,cl2105,cl2110,cl2086,cl2087,cl2077,cl2083,cl2084,cl2085,cl2021,cl2078,cl2079,cl2076,cl2080,cl2081,cl10909,cl2020,cl2013,cl2050,cl2049,cl2048,cl10908,cl2045,cl1062,cl963,cl2015,cl2010,cl2009,cl2012,cl1085,cl1084,cl1050,cl1050b,cl1054,cl1071,cl1059,cl1058,cl987,cl1057,cl1070,cl1069,cl1068,cl1067,cl1066,cl1065,cl1064,cl1063,cl1053,cl1052,cl1049,cl1048,cl1051,cl1051b,cl605,cl611,cl612,cl604,cl613,cl614,cl606,cl917,cl618,cl678,cl988,cl942,cl941,cl937,cl940,cl936,cl958,cl991,cl990,cl964,cl965,cl10906,cl966,cl939,cl916,cl914,cl901,cl935,cl923,cl925,cl900,cl932,cl930,cl619,cl620,cl664,cl617,cl21,cl22,cl23,cl926,cl927,cl928,cl929,cl10900,cl10902,cl10903,cl1076a,cl1918,cl2052,cl20,cl1942,cl12021,cl10912,cl12078,cl12115,cl12079,cl12081,cl11061,cl12080
Exception in thread "main" org.apache.solr.common.SolrException: Can't load schema /data/solr/solr-create/biocache/conf/schema.xml: java.io.IOException: Is a directory
at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:608)
at org.apache.solr.schema.IndexSchema.<init>(IndexSchema.java:182)
at org.apache.solr.schema.ManagedIndexSchema.<init>(ManagedIndexSchema.java:104)
at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:173)
at org.apache.solr.schema.ManagedIndexSchemaFactory.create(ManagedIndexSchemaFactory.java:45)
at org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:75)
at au.org.ala.biocache.index.IndexLocalNode.setUpSolrConfig(IndexLocalNode.scala:341)
at au.org.ala.biocache.index.IndexLocalNode.indexRecords(IndexLocalNode.scala:76)
at au.org.ala.biocache.tool.IndexLocalRecordsV2$.main(IndexLocalRecordsV2.scala:88)
at au.org.ala.biocache.cmd.CMD2$.main(CMD2.scala:130)
at au.org.ala.biocache.cmd.CMD2.main(CMD2.scala)
Caused by: org.apache.solr.common.SolrException: java.io.IOException: Is a directory
at org.apache.solr.schema.ManagedIndexSchema.informResourceLoaderAwareObjectsInChain(ManagedIndexSchema.java:1316)
at org.apache.solr.schema.ManagedIndexSchema.informResourceLoaderAwareObjectsForFieldType(ManagedIndexSchema.java:1167)
at org.apache.solr.schema.ManagedIndexSchema.postReadInform(ManagedIndexSchema.java:1152)
at org.apache.solr.schema.IndexSchema.readSchema(IndexSchema.java:604)
... 10 more
Caused by: java.io.IOException: Is a directory
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at org.apache.lucene.analysis.WordlistLoader.getLines(WordlistLoader.java:217)
at org.apache.lucene.analysis.util.AbstractAnalysisFactory.getLines(AbstractAnalysisFactory.java:254)
at org.apache.lucene.analysis.util.AbstractAnalysisFactory.getWordSet(AbstractAnalysisFactory.java:243)
at org.apache.lucene.analysis.core.StopFilterFactory.inform(StopFilterFactory.java:96)
at org.apache.solr.schema.ManagedIndexSchema.informResourceLoaderAwareObjectsInChain(ManagedIndexSchema.java:1314)
... 13 more
Three of the entries in both of the conf directories are directories instead of files, protwords.txt
, stopwords.txt
and synonyms.txt
.
Those files in zookeeper were 0-byte sized entries and that seems to have been why they were being created on disk as directories rather than files. I modified them for the current biocache collection (biocache-2019-03-06-44
)in zookeeper to contain only the apache commented headers from the examples in biocache-store/src/test/resources/solr/biocache/conf/
and they are now files on disk and the index process has proceeded past where it was failing.
Closing this as fixed as it is now downloading the configuration files to disk correctly from zookeeper. Further comments will be in #315 or others as necessary.
The advice given to get a solr schema to add new fields was to delete the conf directory. The command that was used to remove the directory was:
Doing this causes an error that appears to include a hardcoded solr collection name (or one that is in a configuration file that I haven't looked at):