search

ScaleUnlimited / cascading.solr

Cascading scheme for Solr
27 stars 13 forks source link

Upgrade cascading.solr to support Solr 4.10.1 and Cascading 2.5.6 #7

Closed erasmas closed 10 years ago

erasmas commented 10 years ago

Hi guys, Currently it's not possible to use cascading.solr with Solr 4.10.1 due to API changes in solr-core. I made an attempt to see what needs to be changed in order to support Solr 4.10.1. If you plug in solr-core 4.10.1 you need to change the way you initialize CoreContainer:

solr-core 4.2.1

System.setProperty("solr.solr.home", SolrSchemeUtil.makeTempSolrHome(solrCoreDir).getAbsolutePath());
CoreContainer.Initializer initializer = new CoreContainer.Initializer();
_coreContainer = initializer.initialize();

solr-core 4.10.1

final String solrHome = SolrSchemeUtil.makeTempSolrHome(solrCoreDir).getAbsolutePath();
_coreContainer = new CoreContainer(solrHome);
_coreContainer.load();

This compiles with solr-core 4.10.1 but now I have a test failure and I'm not sure how to fix it:

14/10/13 14:55:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/10/13 14:55:55 INFO util.Hadoop18TapUtil: setting up task: 'attempt_002147483647_0000_m_000000_0' - file:/Users/dmi3y/dev/projects/throwaway/cascading.solr/build/test/SolrSchemeHadoopTest/testSimpleIndexing/in/_temporary/_attempt_002147483647_0000_m_000000_0
14/10/13 14:55:55 INFO hadoop.TupleSerialization: using hadoop serializations from the job conf: cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,cascading.tuple.hadoop.BytesSerialization 
14/10/13 14:55:55 INFO hadoop.TupleSerialization: adding serialization token: 127, for classname: org.apache.hadoop.io.BytesWritable
14/10/13 14:55:55 INFO hadoop.TupleSerialization: adding serialization token: 126, for classname: [B
14/10/13 14:55:55 INFO io.TapOutputCollector: closing tap collector for: build/test/SolrSchemeHadoopTest/testSimpleIndexing/in/part-00000
14/10/13 14:55:55 INFO util.Hadoop18TapUtil: committing task: 'attempt_002147483647_0000_m_000000_0' - file:/Users/dmi3y/dev/projects/throwaway/cascading.solr/build/test/SolrSchemeHadoopTest/testSimpleIndexing/in/_temporary/_attempt_002147483647_0000_m_000000_0
14/10/13 14:55:55 INFO util.Hadoop18TapUtil: saved output of task 'attempt_002147483647_0000_m_000000_0' to file:/Users/dmi3y/dev/projects/throwaway/cascading.solr/build/test/SolrSchemeHadoopTest/testSimpleIndexing/in
14/10/13 14:55:55 INFO util.Hadoop18TapUtil: deleting temp path file:/Users/dmi3y/dev/projects/throwaway/cascading.solr/build/test/SolrSchemeHadoopTest/testSimpleIndexing/in/_temporary
14/10/13 14:55:55 INFO core.SolrResourceLoader: new SolrResourceLoader for directory: '/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/0e95d64c-8b6f-416e-9dff-c182428104e5/'
14/10/13 14:55:55 INFO core.ConfigSolr: Loading container configuration from /var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/0e95d64c-8b6f-416e-9dff-c182428104e5/solr.xml
14/10/13 14:55:55 INFO core.CoreContainer: New CoreContainer 1504642150
14/10/13 14:55:55 INFO core.CoreContainer: Loading cores into CoreContainer [instanceDir=/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/0e95d64c-8b6f-416e-9dff-c182428104e5/]
14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting socketTimeout to: 0
14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting urlScheme to: null
14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting connTimeout to: 0
14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting maxConnectionsPerHost to: 20
14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting corePoolSize to: 0
14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting maximumPoolSize to: 2147483647
14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting maxThreadIdleTime to: 5
14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting sizeOfQueue to: -1
14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting fairnessPolicy to: false
14/10/13 14:55:56 INFO update.UpdateShardHandler: Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false
14/10/13 14:55:56 INFO logging.LogWatcher: SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
14/10/13 14:55:56 INFO logging.LogWatcher: Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
14/10/13 14:55:56 INFO core.CoreContainer: Host Name: null
14/10/13 14:55:56 INFO core.SolrResourceLoader: new SolrResourceLoader for directory: '/Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/'
14/10/13 14:55:56 INFO core.SolrConfig: Adding specified lib dirs to ClassLoader
14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../contrib/extraction/lib (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../contrib/extraction/lib).
14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../dist).
14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../contrib/clustering/lib/ (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../contrib/clustering/lib).
14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../dist).
14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../contrib/langid/lib/ (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../contrib/langid/lib).
14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../dist).
14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../contrib/velocity/lib (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../contrib/velocity/lib).
14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../dist).
14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: /total/crap/dir/ignored (resolved as: /total/crap/dir/ignored).
14/10/13 14:55:56 INFO core.SolrConfig: Using Lucene MatchVersion: 4.1.0
14/10/13 14:55:56 INFO core.Config: Loaded SolrConfig: solrconfig.xml
14/10/13 14:55:56 INFO schema.IndexSchema: Reading Solr Schema from schema.xml
14/10/13 14:55:56 INFO schema.IndexSchema: [collection1] Schema name=example
14/10/13 14:55:56 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.IntField]. Please consult documentation how to replace it accordingly.
14/10/13 14:55:56 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.LongField]. Please consult documentation how to replace it accordingly.
14/10/13 14:55:56 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.FloatField]. Please consult documentation how to replace it accordingly.
14/10/13 14:55:56 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.DoubleField]. Please consult documentation how to replace it accordingly.
14/10/13 14:55:56 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.DateField]. Please consult documentation how to replace it accordingly.
14/10/13 14:55:57 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.ThaiWordFilterFactory]. Please consult documentation how to replace it accordingly.
14/10/13 14:55:57 INFO schema.IndexSchema: unique key field: id
14/10/13 14:55:57 INFO schema.FileExchangeRateProvider: Reloading exchange rates from file currency.xml
14/10/13 14:55:57 INFO schema.FileExchangeRateProvider: Reloading exchange rates from file currency.xml
14/10/13 14:55:57 INFO core.CoreContainer: Creating SolrCore 'collection1' using configuration from instancedir /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/
14/10/13 14:55:57 INFO core.SolrCore: solr.NRTCachingDirectoryFactory
14/10/13 14:55:57 INFO core.SolrCore: [collection1] Opening new SolrCore at /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/, dataDir=/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/
14/10/13 14:55:57 INFO core.JmxMonitoredMap: No JMX servers found, not exposing Solr information with JMX.
14/10/13 14:55:57 INFO core.SolrCore: [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
14/10/13 14:55:57 INFO core.SolrCore: [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]}
14/10/13 14:55:57 INFO core.CachingDirectoryFactory: return new directory for /private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667
14/10/13 14:55:57 INFO core.SolrCore: New index directory detected: old=null new=/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index/
14/10/13 14:55:57 WARN core.SolrCore: [collection1] Solr index directory '/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index' doesn't exist. Creating new index...
14/10/13 14:55:57 INFO core.CachingDirectoryFactory: return new directory for /private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index
14/10/13 14:55:57 INFO core.SolrCore: SolrDeletionPolicy.onCommit: commits: num=1
    commit{dir=NRTCachingDirectory(MMapDirectory@/private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index lockFactory=NativeFSLockFactory@/private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
14/10/13 14:55:57 INFO core.SolrCore: newest commit generation = 1
14/10/13 14:55:57 INFO core.SolrCore: created json: solr.JSONResponseWriter
14/10/13 14:55:57 INFO core.SolrCore: adding lazy queryResponseWriter: solr.VelocityResponseWriter
14/10/13 14:55:57 INFO core.SolrCore: created velocity: solr.VelocityResponseWriter
14/10/13 14:55:57 INFO core.SolrCore: created xslt: solr.XSLTResponseWriter
14/10/13 14:55:57 INFO response.XSLTResponseWriter: xsltCacheLifetimeSeconds=5
14/10/13 14:55:57 INFO core.SolrCore: no updateRequestProcessorChain defined as default, creating implicit default
14/10/13 14:55:58 INFO core.RequestHandlers: created /update: org.apache.solr.handler.UpdateRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /update/json: org.apache.solr.handler.UpdateRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /update/csv: org.apache.solr.handler.UpdateRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /select: solr.SearchHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /query: solr.SearchHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /get: solr.RealTimeGetHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /browse: solr.SearchHandler
14/10/13 14:55:58 WARN core.RequestHandlers: Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /update: solr.UpdateRequestHandler
14/10/13 14:55:58 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.JsonUpdateRequestHandler]. Please consult documentation how to replace it accordingly.
14/10/13 14:55:58 WARN core.RequestHandlers: Multiple requestHandler registered to the same name: /update/json ignoring: org.apache.solr.handler.UpdateRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /update/json: solr.JsonUpdateRequestHandler
14/10/13 14:55:58 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.CSVRequestHandler]. Please consult documentation how to replace it accordingly.
14/10/13 14:55:58 WARN core.RequestHandlers: Multiple requestHandler registered to the same name: /update/csv ignoring: org.apache.solr.handler.UpdateRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /update/csv: solr.CSVRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: adding lazy requestHandler: solr.extraction.ExtractingRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /update/extract: solr.extraction.ExtractingRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: adding lazy requestHandler: solr.FieldAnalysisRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /analysis/field: solr.FieldAnalysisRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: adding lazy requestHandler: solr.DocumentAnalysisRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /analysis/document: solr.DocumentAnalysisRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /admin/: solr.admin.AdminHandlers
14/10/13 14:55:58 INFO core.RequestHandlers: created /admin/ping: solr.PingRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /debug/dump: solr.DumpRequestHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /replication: solr.ReplicationHandler
14/10/13 14:55:58 INFO core.RequestHandlers: adding lazy requestHandler: solr.SearchHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /spell: solr.SearchHandler
14/10/13 14:55:58 INFO core.RequestHandlers: adding lazy requestHandler: solr.SearchHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /tvrh: solr.SearchHandler
14/10/13 14:55:58 INFO core.RequestHandlers: adding lazy requestHandler: solr.SearchHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /terms: solr.SearchHandler
14/10/13 14:55:58 INFO core.RequestHandlers: adding lazy requestHandler: solr.SearchHandler
14/10/13 14:55:58 INFO core.RequestHandlers: created /elevate: solr.SearchHandler
14/10/13 14:55:58 INFO loader.XMLLoader: xsltCacheLifetimeSeconds=60
14/10/13 14:55:58 INFO loader.XMLLoader: xsltCacheLifetimeSeconds=60
14/10/13 14:55:58 INFO loader.XMLLoader: xsltCacheLifetimeSeconds=60
14/10/13 14:55:58 INFO loader.XMLLoader: xsltCacheLifetimeSeconds=60
14/10/13 14:55:58 INFO loader.XMLLoader: xsltCacheLifetimeSeconds=60
14/10/13 14:55:58 INFO loader.XMLLoader: xsltCacheLifetimeSeconds=60
14/10/13 14:55:58 INFO loader.XMLLoader: xsltCacheLifetimeSeconds=60
14/10/13 14:55:58 INFO core.SolrCore: Hard AutoCommit: if uncommited for 15000ms; 
14/10/13 14:55:58 INFO core.SolrCore: Soft AutoCommit: disabled
14/10/13 14:55:58 INFO core.SolrCore: SolrDeletionPolicy.onInit: commits: num=1
    commit{dir=NRTCachingDirectory(MMapDirectory@/private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index lockFactory=NativeFSLockFactory@/private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
14/10/13 14:55:58 INFO core.SolrCore: newest commit generation = 1
14/10/13 14:55:58 INFO search.SolrIndexSearcher: Opening Searcher@7a9dbe7e[collection1] main
14/10/13 14:55:58 INFO rest.ManagedResourceStorage: File-based storage initialized to use dir: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/conf
14/10/13 14:55:58 INFO rest.RestManager: Initializing RestManager with initArgs: {storageDir=/Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/conf}
14/10/13 14:55:58 INFO rest.ManagedResourceStorage: Reading _rest_managed.json using file:dir=/Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/conf
14/10/13 14:55:58 INFO rest.ManagedResourceStorage: Loaded LinkedHashMap at path _rest_managed.json using file:dir=/Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/conf
14/10/13 14:55:58 INFO rest.ManagedResource: Loaded initArgs {} for /rest/managed
14/10/13 14:55:58 INFO rest.RestManager: Initializing 0 registered ManagedResources
14/10/13 14:55:58 INFO component.SpellCheckComponent: Initializing spell checkers
14/10/13 14:55:58 INFO spelling.DirectSolrSpellChecker: init: {name=default,field=name,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
14/10/13 14:55:58 INFO component.SpellCheckComponent: No queryConverter defined, using default converter
14/10/13 14:55:58 INFO component.QueryElevationComponent: Loading QueryElevation from: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/conf/elevate.xml
14/10/13 14:55:58 INFO handler.ReplicationHandler: Commits will be reserved for  10000
14/10/13 14:55:58 INFO core.SolrCore: QuerySenderListener sending requests to Searcher@7a9dbe7e[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
14/10/13 14:55:58 INFO core.CoreContainer: registering core: collection1
14/10/13 14:55:58 INFO core.CoreContainer: Shutting down CoreContainer instance=1504642150
14/10/13 14:55:58 INFO core.SolrCore: [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@f8ed89d
14/10/13 14:55:58 INFO update.UpdateHandler: closing DirectUpdateHandler2{commits=0,autocommit maxTime=15000ms,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
14/10/13 14:55:58 INFO update.SolrCoreState: Closing SolrCoreState
14/10/13 14:55:58 INFO update.DefaultSolrCoreState: SolrCoreState ref count has reached 0 - closing IndexWriter
14/10/13 14:55:58 INFO update.DefaultSolrCoreState: closing IndexWriter with IndexWriterCloser
14/10/13 14:55:58 INFO core.SolrCore: [collection1] webapp=null path=null params={q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false&event=firstSearcher} hits=0 status=0 QTime=83 
14/10/13 14:55:58 INFO core.SolrCore: QuerySenderListener done.
14/10/13 14:55:58 INFO component.SpellCheckComponent: Loading spell index for spellchecker: default
14/10/13 14:55:58 INFO component.SpellCheckComponent: Loading spell index for spellchecker: wordbreak
14/10/13 14:55:58 INFO core.SolrCore: [collection1] Registered new searcher Searcher@7a9dbe7e[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
14/10/13 14:55:58 INFO core.SolrCore: [collection1] Closing main searcher on request.
14/10/13 14:55:58 INFO core.CachingDirectoryFactory: Closing NRTCachingDirectoryFactory - 2 directories currently being tracked
14/10/13 14:55:58 INFO core.CachingDirectoryFactory: looking to close /private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667 [CachedDir<<refCount=0;path=/private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667;done=false>>]
14/10/13 14:55:58 INFO core.CachingDirectoryFactory: Closing directory: /private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667
14/10/13 14:55:58 INFO core.CachingDirectoryFactory: looking to close /private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index [CachedDir<<refCount=0;path=/private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index;done=false>>]
14/10/13 14:55:58 INFO core.CachingDirectoryFactory: Closing directory: /private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index
14/10/13 14:55:58 INFO util.HadoopUtil: resolving application jar from found main method on: com.intellij.rt.execution.junit.JUnitStarter
14/10/13 14:55:58 INFO planner.HadoopPlanner: using application jar: /Applications/IntelliJ IDEA 13.app/Contents/plugins/junit/lib/junit-rt.jar
14/10/13 14:55:58 INFO property.AppProps: using app.id: FCDE80B656C063F23B469E9254D958F1
14/10/13 14:55:59 INFO util.Version: Concurrent, Inc - Cascading 2.1.6
14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr] starting
14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr]  source: Hfs["SequenceFile[['id', 'name', 'price', 'cat', 'inStock', 'image']]"]["build/test/SolrSchemeHadoopTest/testSimpleIndexing/in"]
14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr]  sink: Hfs["SolrScheme[['id', 'name', 'price', 'cat', 'inStock', 'image']]"]["build/test/SolrSchemeHadoopTest/testSimpleIndexing/out"]
14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr]  parallel execution is enabled: false
14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr]  starting jobs: 1
14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr]  allocating threads: 1
14/10/13 14:55:59 INFO flow.FlowStep: [tuples to Solr] starting step: (1/1) ...st/testSimpleIndexing/out
14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr] stopping all jobs
14/10/13 14:55:59 INFO flow.FlowStep: [tuples to Solr] stopping: (1/1) ...st/testSimpleIndexing/out
14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr] stopped all jobs

cascading.flow.FlowException: unhandled exception
    at cascading.flow.BaseFlow.complete(BaseFlow.java:825)
    at com.scaleunlimited.cascading.scheme.core.AbstractSolrSchemeTest.testSimpleIndexing(AbstractSolrSchemeTest.java:144)
    at com.scaleunlimited.cascading.scheme.hadoop.SolrSchemeHadoopTest.testSimpleIndexing(SolrSchemeHadoopTest.java:100)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    org/apache/hadoop/mapred/JobTrackerInstrumentation.create(Lorg/apache/hadoop/mapred/JobTracker;Lorg/apache/hadoop/mapred/JobConf;)Lorg/apache/hadoop/mapred/JobTrackerInstrumentation; @5: invokestatic
  Reason:
    Type 'org/apache/hadoop/metrics2/lib/DefaultMetricsSystem' (current frame, stack[2]) is not assignable to 'org/apache/hadoop/metrics2/MetricsSystem'
  Current Frame:
    bci: @5
    flags: { }
    locals: { 'org/apache/hadoop/mapred/JobTracker', 'org/apache/hadoop/mapred/JobConf' }
    stack: { 'org/apache/hadoop/mapred/JobTracker', 'org/apache/hadoop/mapred/JobConf', 'org/apache/hadoop/metrics2/lib/DefaultMetricsSystem' }
  Bytecode:
    0000000: 2a2b b200 03b8 0004 b0                 

    at org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:420)
    at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472)
    at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:457)
    at cascading.flow.hadoop.planner.HadoopFlowStepJob.internalNonBlockingStart(HadoopFlowStepJob.java:103)
    at cascading.flow.planner.FlowStepJob.blockOnJob(FlowStepJob.java:191)
    at cascading.flow.planner.FlowStepJob.start(FlowStepJob.java:145)
    at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:120)
    at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:42)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)

I've also noticed that upgrading hadoop-core to 2.5.1 and cascading to 2.5.6 requires additional changes. I would appreciate if you can help upgrading cascading.solr since it's quite useful in both Cascading and Cascalog applications.

bixolabs commented 10 years ago

Hi Dmitriy,

Thanks for giving this a try.

I'd run into the same core container initialization issue.

What Solr home (schema, solrconfig, etc) are you trying to use this with? I see a lot of deprecated class warnings, etc.

I've also noticed that upgrading hadoop-core to 2.5.1 and cascading to 2.5.6 requires additional changes

What changes?

Thanks,

-- Ken

From: Dmitriy Morozov Sent: October 13, 2014 5:09:44am PDT To: ScaleUnlimited/cascading.solr Subject: [cascading.solr] Upgrade cascading.solr to support Solr 4.10.1 and Cascading 2.5.6 (#7)

Hi guys, Currently it's not possible to use cascading.solr with Solr 4.10.1 due to API changes in solr-core. I made an attempt to see what needs to be changed in order to support Solr 4.10.1. If you plug in solr-core 4.10.1 you need to change the way you initialize CoreContainer:

solr-core 4.2.1

System.setProperty("solr.solr.home", SolrSchemeUtil.makeTempSolrHome(solrCoreDir).getAbsolutePath()); CoreContainer.Initializer initializer = new CoreContainer.Initializer(); _coreContainer = initializer.initialize(); solr-core 4.10.1

final String solrHome = SolrSchemeUtil.makeTempSolrHome(solrCoreDir).getAbsolutePath(); _coreContainer = new CoreContainer(solrHome); _coreContainer.load(); This compiles with solr-core 4.10.1 but now I have a test failure and I'm not sure how to fix it:

14/10/13 14:55:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 14/10/13 14:55:55 INFO util.Hadoop18TapUtil: setting up task: 'attempt_002147483647_0000_m_000000_0' - file:/Users/dmi3y/dev/projects/throwaway/cascading.solr/build/test/SolrSchemeHadoopTest/testSimpleIndexing/in/_temporary/_attempt_002147483647_0000_m_000000_0 14/10/13 14:55:55 INFO hadoop.TupleSerialization: using hadoop serializations from the job conf: cascading.tuple.hadoop.TupleSerialization,org.apache.hadoop.io.serializer.WritableSerialization,cascading.tuple.hadoop.BytesSerialization 14/10/13 14:55:55 INFO hadoop.TupleSerialization: adding serialization token: 127, for classname: org.apache.hadoop.io.BytesWritable 14/10/13 14:55:55 INFO hadoop.TupleSerialization: adding serialization token: 126, for classname: [B 14/10/13 14:55:55 INFO io.TapOutputCollector: closing tap collector for: build/test/SolrSchemeHadoopTest/testSimpleIndexing/in/part-00000 14/10/13 14:55:55 INFO util.Hadoop18TapUtil: committing task: 'attempt_002147483647_0000_m_000000_0' - file:/Users/dmi3y/dev/projects/throwaway/cascading.solr/build/test/SolrSchemeHadoopTest/testSimpleIndexing/in/_temporary/_attempt_002147483647_0000_m_000000_0 14/10/13 14:55:55 INFO util.Hadoop18TapUtil: saved output of task 'attempt_002147483647_0000_m_000000_0' to file:/Users/dmi3y/dev/projects/throwaway/cascading.solr/build/test/SolrSchemeHadoopTest/testSimpleIndexing/in 14/10/13 14:55:55 INFO util.Hadoop18TapUtil: deleting temp path file:/Users/dmi3y/dev/projects/throwaway/cascading.solr/build/test/SolrSchemeHadoopTest/testSimpleIndexing/in/_temporary 14/10/13 14:55:55 INFO core.SolrResourceLoader: new SolrResourceLoader for directory: '/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/0e95d64c-8b6f-416e-9dff-c182428104e5/' 14/10/13 14:55:55 INFO core.ConfigSolr: Loading container configuration from /var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/0e95d64c-8b6f-416e-9dff-c182428104e5/solr.xml 14/10/13 14:55:55 INFO core.CoreContainer: New CoreContainer 1504642150 14/10/13 14:55:55 INFO core.CoreContainer: Loading cores into CoreContainer [instanceDir=/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/0e95d64c-8b6f-416e-9dff-c182428104e5/] 14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting socketTimeout to: 0 14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting urlScheme to: null 14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting connTimeout to: 0 14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting maxConnectionsPerHost to: 20 14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting corePoolSize to: 0 14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting maximumPoolSize to: 2147483647 14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting maxThreadIdleTime to: 5 14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting sizeOfQueue to: -1 14/10/13 14:55:55 INFO component.HttpShardHandlerFactory: Setting fairnessPolicy to: false 14/10/13 14:55:56 INFO update.UpdateShardHandler: Creating UpdateShardHandler HTTP client with params: socketTimeout=0&connTimeout=0&retry=false 14/10/13 14:55:56 INFO logging.LogWatcher: SLF4J impl is org.slf4j.impl.Log4jLoggerFactory 14/10/13 14:55:56 INFO logging.LogWatcher: Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)] 14/10/13 14:55:56 INFO core.CoreContainer: Host Name: null 14/10/13 14:55:56 INFO core.SolrResourceLoader: new SolrResourceLoader for directory: '/Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/' 14/10/13 14:55:56 INFO core.SolrConfig: Adding specified lib dirs to ClassLoader 14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../contrib/extraction/lib (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../contrib/extraction/lib). 14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../dist). 14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../contrib/clustering/lib/ (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../contrib/clustering/lib). 14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../dist). 14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../contrib/langid/lib/ (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../contrib/langid/lib). 14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../dist). 14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../contrib/velocity/lib (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../contrib/velocity/lib). 14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: ../../../dist/ (resolved as: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/../../../dist). 14/10/13 14:55:56 WARN core.SolrResourceLoader: Can't find (or read) directory to add to classloader: /total/crap/dir/ignored (resolved as: /total/crap/dir/ignored). 14/10/13 14:55:56 INFO core.SolrConfig: Using Lucene MatchVersion: 4.1.0 14/10/13 14:55:56 INFO core.Config: Loaded SolrConfig: solrconfig.xml 14/10/13 14:55:56 INFO schema.IndexSchema: Reading Solr Schema from schema.xml 14/10/13 14:55:56 INFO schema.IndexSchema: [collection1] Schema name=example 14/10/13 14:55:56 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.IntField]. Please consult documentation how to replace it accordingly. 14/10/13 14:55:56 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.LongField]. Please consult documentation how to replace it accordingly. 14/10/13 14:55:56 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.FloatField]. Please consult documentation how to replace it accordingly. 14/10/13 14:55:56 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.DoubleField]. Please consult documentation how to replace it accordingly. 14/10/13 14:55:56 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.DateField]. Please consult documentation how to replace it accordingly. 14/10/13 14:55:57 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.ThaiWordFilterFactory]. Please consult documentation how to replace it accordingly. 14/10/13 14:55:57 INFO schema.IndexSchema: unique key field: id 14/10/13 14:55:57 INFO schema.FileExchangeRateProvider: Reloading exchange rates from file currency.xml 14/10/13 14:55:57 INFO schema.FileExchangeRateProvider: Reloading exchange rates from file currency.xml 14/10/13 14:55:57 INFO core.CoreContainer: Creating SolrCore 'collection1' using configuration from instancedir /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/ 14/10/13 14:55:57 INFO core.SolrCore: solr.NRTCachingDirectoryFactory 14/10/13 14:55:57 INFO core.SolrCore: [collection1] Opening new SolrCore at /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/, dataDir=/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/ 14/10/13 14:55:57 INFO core.JmxMonitoredMap: No JMX servers found, not exposing Solr information with JMX. 14/10/13 14:55:57 INFO core.SolrCore: [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[]} 14/10/13 14:55:57 INFO core.SolrCore: [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=static firstSearcher warming in solrconfig.xml}]} 14/10/13 14:55:57 INFO core.CachingDirectoryFactory: return new directory for /private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667 14/10/13 14:55:57 INFO core.SolrCore: New index directory detected: old=null new=/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index/ 14/10/13 14:55:57 WARN core.SolrCore: [collection1] Solr index directory '/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index' doesn't exist. Creating new index... 14/10/13 14:55:57 INFO core.CachingDirectoryFactory: return new directory for /private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index 14/10/13 14:55:57 INFO core.SolrCore: SolrDeletionPolicy.onCommit: commits: num=1 commit{dir=NRTCachingDirectory(MMapDirectory@/private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index lockFactory=NativeFSLockFactory@/private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1} 14/10/13 14:55:57 INFO core.SolrCore: newest commit generation = 1 14/10/13 14:55:57 INFO core.SolrCore: created json: solr.JSONResponseWriter 14/10/13 14:55:57 INFO core.SolrCore: adding lazy queryResponseWriter: solr.VelocityResponseWriter 14/10/13 14:55:57 INFO core.SolrCore: created velocity: solr.VelocityResponseWriter 14/10/13 14:55:57 INFO core.SolrCore: created xslt: solr.XSLTResponseWriter 14/10/13 14:55:57 INFO response.XSLTResponseWriter: xsltCacheLifetimeSeconds=5 14/10/13 14:55:57 INFO core.SolrCore: no updateRequestProcessorChain defined as default, creating implicit default 14/10/13 14:55:58 INFO core.RequestHandlers: created /update: org.apache.solr.handler.UpdateRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /update/json: org.apache.solr.handler.UpdateRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /update/csv: org.apache.solr.handler.UpdateRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /select: solr.SearchHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /query: solr.SearchHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /get: solr.RealTimeGetHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /browse: solr.SearchHandler 14/10/13 14:55:58 WARN core.RequestHandlers: Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /update: solr.UpdateRequestHandler 14/10/13 14:55:58 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.JsonUpdateRequestHandler]. Please consult documentation how to replace it accordingly. 14/10/13 14:55:58 WARN core.RequestHandlers: Multiple requestHandler registered to the same name: /update/json ignoring: org.apache.solr.handler.UpdateRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /update/json: solr.JsonUpdateRequestHandler 14/10/13 14:55:58 WARN core.SolrResourceLoader: Solr loaded a deprecated plugin/analysis class [solr.CSVRequestHandler]. Please consult documentation how to replace it accordingly. 14/10/13 14:55:58 WARN core.RequestHandlers: Multiple requestHandler registered to the same name: /update/csv ignoring: org.apache.solr.handler.UpdateRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /update/csv: solr.CSVRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: adding lazy requestHandler: solr.extraction.ExtractingRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /update/extract: solr.extraction.ExtractingRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: adding lazy requestHandler: solr.FieldAnalysisRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /analysis/field: solr.FieldAnalysisRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: adding lazy requestHandler: solr.DocumentAnalysisRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /analysis/document: solr.DocumentAnalysisRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /admin/: solr.admin.AdminHandlers 14/10/13 14:55:58 INFO core.RequestHandlers: created /admin/ping: solr.PingRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /debug/dump: solr.DumpRequestHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /replication: solr.ReplicationHandler 14/10/13 14:55:58 INFO core.RequestHandlers: adding lazy requestHandler: solr.SearchHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /spell: solr.SearchHandler 14/10/13 14:55:58 INFO core.RequestHandlers: adding lazy requestHandler: solr.SearchHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /tvrh: solr.SearchHandler 14/10/13 14:55:58 INFO core.RequestHandlers: adding lazy requestHandler: solr.SearchHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /terms: solr.SearchHandler 14/10/13 14:55:58 INFO core.RequestHandlers: adding lazy requestHandler: solr.SearchHandler 14/10/13 14:55:58 INFO core.RequestHandlers: created /elevate: solr.SearchHandler 14/10/13 14:55:58 INFO loader.XMLLoader: xsltCacheLifetimeSeconds=60 14/10/13 14:55:58 INFO loader.XMLLoader: xsltCacheLifetimeSeconds=60 14/10/13 14:55:58 INFO loader.XMLLoader: xsltCacheLifetimeSeconds=60 14/10/13 14:55:58 INFO loader.XMLLoader: xsltCacheLifetimeSeconds=60 14/10/13 14:55:58 INFO loader.XMLLoader: xsltCacheLifetimeSeconds=60 14/10/13 14:55:58 INFO loader.XMLLoader: xsltCacheLifetimeSeconds=60 14/10/13 14:55:58 INFO loader.XMLLoader: xsltCacheLifetimeSeconds=60 14/10/13 14:55:58 INFO core.SolrCore: Hard AutoCommit: if uncommited for 15000ms; 14/10/13 14:55:58 INFO core.SolrCore: Soft AutoCommit: disabled 14/10/13 14:55:58 INFO core.SolrCore: SolrDeletionPolicy.onInit: commits: num=1 commit{dir=NRTCachingDirectory(MMapDirectory@/private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index lockFactory=NativeFSLockFactory@/private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1} 14/10/13 14:55:58 INFO core.SolrCore: newest commit generation = 1 14/10/13 14:55:58 INFO search.SolrIndexSearcher: Opening Searcher@7a9dbe7e[collection1] main 14/10/13 14:55:58 INFO rest.ManagedResourceStorage: File-based storage initialized to use dir: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/conf 14/10/13 14:55:58 INFO rest.RestManager: Initializing RestManager with initArgs: {storageDir=/Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/conf} 14/10/13 14:55:58 INFO rest.ManagedResourceStorage: Reading _rest_managed.json using file:dir=/Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/conf 14/10/13 14:55:58 INFO rest.ManagedResourceStorage: Loaded LinkedHashMap at path _rest_managed.json using file:dir=/Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/conf 14/10/13 14:55:58 INFO rest.ManagedResource: Loaded initArgs {} for /rest/managed 14/10/13 14:55:58 INFO rest.RestManager: Initializing 0 registered ManagedResources 14/10/13 14:55:58 INFO component.SpellCheckComponent: Initializing spell checkers 14/10/13 14:55:58 INFO spelling.DirectSolrSpellChecker: init: {name=default,field=name,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01} 14/10/13 14:55:58 INFO component.SpellCheckComponent: No queryConverter defined, using default converter 14/10/13 14:55:58 INFO component.QueryElevationComponent: Loading QueryElevation from: /Users/dmi3y/dev/projects/throwaway/cascading.solr/src/test/resources/solr-home-4.1/collection1/conf/elevate.xml 14/10/13 14:55:58 INFO handler.ReplicationHandler: Commits will be reserved for 10000 14/10/13 14:55:58 INFO core.SolrCore: QuerySenderListener sending requests to Searcher@7a9dbe7e[collection1] main{StandardDirectoryReader(segments_1:1:nrt)} 14/10/13 14:55:58 INFO core.CoreContainer: registering core: collection1 14/10/13 14:55:58 INFO core.CoreContainer: Shutting down CoreContainer instance=1504642150 14/10/13 14:55:58 INFO core.SolrCore: [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@f8ed89d 14/10/13 14:55:58 INFO update.UpdateHandler: closing DirectUpdateHandler2{commits=0,autocommit maxTime=15000ms,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0} 14/10/13 14:55:58 INFO update.SolrCoreState: Closing SolrCoreState 14/10/13 14:55:58 INFO update.DefaultSolrCoreState: SolrCoreState ref count has reached 0 - closing IndexWriter 14/10/13 14:55:58 INFO update.DefaultSolrCoreState: closing IndexWriter with IndexWriterCloser 14/10/13 14:55:58 INFO core.SolrCore: [collection1] webapp=null path=null params={q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false&event=firstSearcher} hits=0 status=0 QTime=83 14/10/13 14:55:58 INFO core.SolrCore: QuerySenderListener done. 14/10/13 14:55:58 INFO component.SpellCheckComponent: Loading spell index for spellchecker: default 14/10/13 14:55:58 INFO component.SpellCheckComponent: Loading spell index for spellchecker: wordbreak 14/10/13 14:55:58 INFO core.SolrCore: [collection1] Registered new searcher Searcher@7a9dbe7e[collection1] main{StandardDirectoryReader(segments_1:1:nrt)} 14/10/13 14:55:58 INFO core.SolrCore: [collection1] Closing main searcher on request. 14/10/13 14:55:58 INFO core.CachingDirectoryFactory: Closing NRTCachingDirectoryFactory - 2 directories currently being tracked 14/10/13 14:55:58 INFO core.CachingDirectoryFactory: looking to close /private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667 [CachedDir<>] 14/10/13 14:55:58 INFO core.CachingDirectoryFactory: Closing directory: /private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667 14/10/13 14:55:58 INFO core.CachingDirectoryFactory: looking to close /private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index [CachedDir<>] 14/10/13 14:55:58 INFO core.CachingDirectoryFactory: Closing directory: /private/var/folders/jd/5trblx394vn7j1btnq393m9h0000gn/T/7aaaa5cf-f8f9-4708-9d09-71a18199c667/index 14/10/13 14:55:58 INFO util.HadoopUtil: resolving application jar from found main method on: com.intellij.rt.execution.junit.JUnitStarter 14/10/13 14:55:58 INFO planner.HadoopPlanner: using application jar: /Applications/IntelliJ IDEA 13.app/Contents/plugins/junit/lib/junit-rt.jar 14/10/13 14:55:58 INFO property.AppProps: using app.id: FCDE80B656C063F23B469E9254D958F1 14/10/13 14:55:59 INFO util.Version: Concurrent, Inc - Cascading 2.1.6 14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr] starting 14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr] source: Hfs["SequenceFile[['id', 'name', 'price', 'cat', 'inStock', 'image']]"]["build/test/SolrSchemeHadoopTest/testSimpleIndexing/in"] 14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr] sink: Hfs["SolrScheme[['id', 'name', 'price', 'cat', 'inStock', 'image']]"]["build/test/SolrSchemeHadoopTest/testSimpleIndexing/out"] 14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr] parallel execution is enabled: false 14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr] starting jobs: 1 14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr] allocating threads: 1 14/10/13 14:55:59 INFO flow.FlowStep: [tuples to Solr] starting step: (1/1) ...st/testSimpleIndexing/out 14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr] stopping all jobs 14/10/13 14:55:59 INFO flow.FlowStep: [tuples to Solr] stopping: (1/1) ...st/testSimpleIndexing/out 14/10/13 14:55:59 INFO flow.Flow: [tuples to Solr] stopped all jobs

cascading.flow.FlowException: unhandled exception at cascading.flow.BaseFlow.complete(BaseFlow.java:825) at com.scaleunlimited.cascading.scheme.core.AbstractSolrSchemeTest.testSimpleIndexing(AbstractSolrSchemeTest.java:144) at com.scaleunlimited.cascading.scheme.hadoop.SolrSchemeHadoopTest.testSimpleIndexing(SolrSchemeHadoopTest.java:100) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.junit.runner.JUnitCore.run(JUnitCore.java:157) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) Caused by: java.lang.VerifyError: Bad type on operand stack Exception Details: Location: org/apache/hadoop/mapred/JobTrackerInstrumentation.create(Lorg/apache/hadoop/mapred/JobTracker;Lorg/apache/hadoop/mapred/JobConf;)Lorg/apache/hadoop/mapred/JobTrackerInstrumentation; @5: invokestatic Reason: Type 'org/apache/hadoop/metrics2/lib/DefaultMetricsSystem' (current frame, stack[2]) is not assignable to 'org/apache/hadoop/metrics2/MetricsSystem' Current Frame: bci: @5 flags: { } locals: { 'org/apache/hadoop/mapred/JobTracker', 'org/apache/hadoop/mapred/JobConf' } stack: { 'org/apache/hadoop/mapred/JobTracker', 'org/apache/hadoop/mapred/JobConf', 'org/apache/hadoop/metrics2/lib/DefaultMetricsSystem' } Bytecode: 0000000: 2a2b b200 03b8 0004 b0 

at org.apache.hadoop.mapred.LocalJobRunner.<init>(LocalJobRunner.java:420)
at org.apache.hadoop.mapred.JobClient.init(JobClient.java:472)
at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:457)
at cascading.flow.hadoop.planner.HadoopFlowStepJob.internalNonBlockingStart(HadoopFlowStepJob.java:103)
at cascading.flow.planner.FlowStepJob.blockOnJob(FlowStepJob.java:191)
at cascading.flow.planner.FlowStepJob.start(FlowStepJob.java:145)
at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:120)
at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:42)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)

I've also noticed that upgrading hadoop-core to 2.5.1 and cascading to 2.5.6 requires additional changes.

— Reply to this email directly or view it on GitHub.

Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr

Ken Krugler +1 530-210-6378 http://www.scaleunlimited.com custom big data solutions & training Hadoop, Cascading, Cassandra & Solr

erasmas commented 10 years ago

What Solr home (schema, solrconfig, etc) are you trying to use this with?

I use schema and config provided with Solr 4.10.1 (solr-4.10.1/example/solr/collection1).

What changes?

I think hadoop-core has changed it's API significantly in 1.2.1. There were many broken imports when I upgraded it to 1.2.1 in cascading.solr.

Thank you.

erasmas commented 10 years ago

Hi, guys. Sorry for bugging you with this issue, but I still have hope that you can help me upgrade cascading.solr to support latest cascading and solr libraries. I forked the project and did few changes mostly upgrading libraries but there two unit tests that I don't know how to fix. Those are testSimpleIndexing for SolrSchemeLocalTest and SolrSchemeHadoopTest.

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.scaleunlimited.cascading.scheme.hadoop.SolrSchemeHadoopTest
log4j:WARN No appenders could be found for logger (org.apache.solr.core.SolrResourceLoader).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 9.902 sec <<< FAILURE!
testSimpleIndexing(com.scaleunlimited.cascading.scheme.hadoop.SolrSchemeHadoopTest)  Time elapsed: 3.566 sec  <<< ERROR!
cascading.flow.FlowException: unhandled exception
    at cascading.flow.BaseFlow.complete(BaseFlow.java:894)
    at com.scaleunlimited.cascading.scheme.core.AbstractSolrSchemeTest.testSimpleIndexing(AbstractSolrSchemeTest.java:145)
    at com.scaleunlimited.cascading.scheme.hadoop.SolrSchemeHadoopTest.testSimpleIndexing(SolrSchemeHadoopTest.java:98)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
    at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
    at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
    at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
    at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
    at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
Caused by: java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
    at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
    at org.apache.hadoop.mapred.JobClient.init(JobClient.java:470)
    at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:449)
    at cascading.flow.hadoop.planner.HadoopFlowStepJob.internalNonBlockingStart(HadoopFlowStepJob.java:106)
    at cascading.flow.planner.FlowStepJob.blockOnJob(FlowStepJob.java:196)
    at cascading.flow.planner.FlowStepJob.start(FlowStepJob.java:149)
    at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:124)
    at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:43)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)

Running com.scaleunlimited.cascading.scheme.local.SolrSchemeLocalTest
Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.986 sec <<< FAILURE!
testSimpleIndexing(com.scaleunlimited.cascading.scheme.local.SolrSchemeLocalTest)  Time elapsed: 0.742 sec  <<< ERROR!
cascading.flow.FlowException: local step failed
    at cascading.flow.planner.FlowStepJob.blockOnJob(FlowStepJob.java:219)
    at cascading.flow.planner.FlowStepJob.start(FlowStepJob.java:149)
    at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:124)
    at cascading.flow.planner.FlowStepJob.call(FlowStepJob.java:43)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)
Caused by: cascading.tuple.TupleException: unable to read from input identifier: build/test/SolrSchemeLocalTest/testSimpleIndexing/in
    at cascading.tuple.TupleEntrySchemeIterator.hasNext(TupleEntrySchemeIterator.java:127)
    at cascading.flow.stream.SourceStage.map(SourceStage.java:76)
    at cascading.flow.stream.SourceStage.call(SourceStage.java:53)
    at cascading.flow.stream.SourceStage.call(SourceStage.java:38)
    ... 4 more
Caused by: java.lang.IllegalArgumentException: current entry is modifiable but given tuple is not modifiable, make copy of given Tuple first
    at cascading.tuple.TupleEntry.setTuple(TupleEntry.java:367)
    at com.scaleunlimited.cascading.local.KryoScheme.source(KryoScheme.java:67)
    at cascading.tuple.TupleEntrySchemeIterator.getNext(TupleEntrySchemeIterator.java:140)
    at cascading.tuple.TupleEntrySchemeIterator.hasNext(TupleEntrySchemeIterator.java:120)
    ... 7 more

Results :

Tests in error:
  testSimpleIndexing(com.scaleunlimited.cascading.scheme.hadoop.SolrSchemeHadoopTest): unhandled exception
  testSimpleIndexing(com.scaleunlimited.cascading.scheme.local.SolrSchemeLocalTest): local step failed

Tests run: 12, Failures: 0, Errors: 2, Skipped: 0

My assumption is that it's the unit tests that need to be updated, there's no regression, since I was able to write Solr index using Cascading workflow and example schema from Solr 4.10.1. I would appreciate guys if you could have a look and provide any feedback on how those tests can be fixed. Thanks!

kkrugler commented 10 years ago

I spent a few hours of fun digging into these issues. The Hadoop one is solved (missing dependencies from Hadoop poms, arghh) but there's an upstream issue with cascading.utils and its KryoScheme, which is used in the unit test when running in local mode. I've pinged Chris Wensel to see what's up, and once that's fixed we can do a new release of cascading.utils, and then cascading.solr.

cwensel commented 10 years ago

the short term fix looks like you should not be serializing the Tuple#isUnmodifiable value.

If there are no test failures, i'll make this transient in 2.6.1

On Oct 25, 2014, at 3:22 PM, Ken Krugler notifications@github.com wrote:

I spent a few hours of fun digging into these issues. The Hadoop one is solved (missing dependencies from Hadoop poms, arghh) but there's an upstream issue with cascading.utils and its KryoScheme, which is used in the unit test when running in local mode. I've pinged Chris Wensel to see what's up, and once that's fixed we can do a new release of cascading.utils, and then cascading.solr.

— Reply to this email directly or view it on GitHub.

Chris K Wensel chris@concurrentinc.com http://concurrentinc.com

kkrugler commented 10 years ago

OK - just released cascading.solr 2.5.0, which depends on cascading.utils 2.5.1 for tests and some additional scaffolding.

erasmas commented 10 years ago

Awesome, thanks a lot!