search

USCDataScience / sparkler

Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
http://irds.usc.edu/sparkler/
Apache License 2.0
411 stars 143 forks source link

Error when injecting urls #124

Closed User12300 closed 6 years ago

User12300 commented 7 years ago

I am new to Sparkler. I begun with the requirements provided and everything worked well. Then, when I started to inject some urls, I got an issue.

Here are the steps that I followed to inject urls:

1) cd to the root directory of the project 2) docker run -it sparkler-local 3) /data/solr/bin/solr 4) /data/sparkler/bin/sparkler.sh 5) java -jar sparkler-app-0.1-SNAPSHOT.jar inject -sf seed.txt

The output that it must appear is:

2016-06-07 19:22:49 INFO  Injector$:70 [main] - Injecting 2 seeds
>>jobId = sparkler-job-1465352569649

But, I got this output: Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar

I don't understand what caused this problem and how can I resolve it. I will be very gratefull if I get an answer. I look forward to your reply.

Thank you.

buggtb commented 7 years ago

I've not run the docker container, but basically you're not providing a path to the jar so it can't find it, either provide the path or I suspect sparkler.sh does the job for you.

java -jar /data/sparkler/...... I guess is what you're after.

Tom

On Fri, Aug 4, 2017 at 10:58 PM, User12300 notifications@github.com wrote:

I am new to Sparkler. I begun with the requirements provided and everything worked well. Then, when I started to inject some urls, I got an issue.

Here are the steps that I followed to inject urls:

  1. cd to the root directory of the project
  2. docker run -it sparkler-local
  3. /data/solr/bin/solr
  4. /data/sparkler/bin/sparkler.sh
  5. java -jar sparkler-app-0.1-SNAPSHOT.jar inject -sf seed.txt

The output that it must appear is:

2016-06-07 19:22:49 INFO Injector$:70 [main] - Injecting 2 seeds

jobId = sparkler-job-1465352569649

But, I got this output: Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar

I don't understand what caused this problem and how can I resolve it. I will be very gratefull if I get an answer. I look forward to your reply.

Thank you.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/USCDataScience/sparkler/issues/124, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGUeG0IrrlWm_X2B4tabPpv9UkQ87RBks5sU5QLgaJpZM4OuO08 .

-- Tom Barber CTO Spicule LTD tom@spicule.co.uk

http://spicule.co.uk

@spiculeim http://twitter.com/spiculeim

Schedule a meeting with me http://meetme.so/spicule

GB: +44(0)5603641316 US: +18448141689

https://leanpub.com/juju-cookbook

User12300 commented 7 years ago

Thank you for your reply. I tried to add the path to the jar but it still doesn't work unfortunately. When I use sparkler.sh it shows this output: sparkler.sh: command not found

So, I tried this: bash sparkler.sh and the output was: 

Sub Commands:
  inject : edu.usc.irds.sparkler.service.Injector 
         - Inject (seed) URLS to crawldb
   crawl : edu.usc.irds.sparkler.pipeline.Crawler 
         - Run crawl pipeline for several iterations

And when I continue to inject, it shows the same error: Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar

buggtb commented 7 years ago

java -jar /data/sparkler/sparkler-app/target/sparkler-app-0.1-SNAPSHOT.jar

is what you're after. Although it seems to me like solr installed with the wrong permissions currently when i built it, I'll fix that.

Tom

On Sat, Aug 5, 2017 at 6:09 PM, User12300 notifications@github.com wrote:

Thank you for your reply. I tried to add the path to the jar but it still doesn't work unfortunately. When I use sparkler.sh it shows this output: sparkler.sh: command not found

So, I tried this: bash sparkler.sh and the output was:

Sub Commands: inject : edu.usc.irds.sparkler.service.Injector

  • Inject (seed) URLS to crawldb crawl : edu.usc.irds.sparkler.pipeline.Crawler
  • Run crawl pipeline for several iterations

And when I continue to inject, it shows the same error: Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/USCDataScience/sparkler/issues/124#issuecomment-320455398, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGUeJzJxEj4x6AkzFSx-1Z0qiEsYUkxks5sVKHVgaJpZM4OuO08 .

-- Tom Barber CTO Spicule LTD tom@spicule.co.uk

http://spicule.co.uk

@spiculeim http://twitter.com/spiculeim

Schedule a meeting with me http://meetme.so/spicule

GB: +44(0)5603641316 US: +18448141689

https://leanpub.com/juju-cookbook

buggtb commented 7 years ago

As you'll see here it works now: https://asciinema.org/a/2tslGQY2MGyaGlVy8rEU0fohC

The only change I have to make is cd to /data/sparkler before running the crawl.

Tom

On Sun, Aug 6, 2017 at 3:32 AM, Tom Barber tom@spicule.co.uk wrote:

java -jar /data/sparkler/sparkler-app/target/sparkler-app-0.1- SNAPSHOT.jar

is what you're after. Although it seems to me like solr installed with the wrong permissions currently when i built it, I'll fix that.

Tom

On Sat, Aug 5, 2017 at 6:09 PM, User12300 notifications@github.com wrote:

Thank you for your reply. I tried to add the path to the jar but it still doesn't work unfortunately. When I use sparkler.sh it shows this output: sparkler.sh: command not found

So, I tried this: bash sparkler.sh and the output was:

Sub Commands: inject : edu.usc.irds.sparkler.service.Injector

  • Inject (seed) URLS to crawldb crawl : edu.usc.irds.sparkler.pipeline.Crawler
  • Run crawl pipeline for several iterations

And when I continue to inject, it shows the same error: Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/USCDataScience/sparkler/issues/124#issuecomment-320455398, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGUeJzJxEj4x6AkzFSx-1Z0qiEsYUkxks5sVKHVgaJpZM4OuO08 .

-- Tom Barber CTO Spicule LTD tom@spicule.co.uk

http://spicule.co.uk

@spiculeim http://twitter.com/spiculeim

Schedule a meeting with me http://meetme.so/spicule

GB: +44(0)5603641316 <+44%2056%200364%201316> US: +18448141689 <(844)%20814-1689>

https://leanpub.com/juju-cookbook

-- Tom Barber CTO Spicule LTD tom@spicule.co.uk

http://spicule.co.uk

@spiculeim http://twitter.com/spiculeim

Schedule a meeting with me http://meetme.so/spicule

GB: +44(0)5603641316 US: +18448141689

https://leanpub.com/juju-cookbook

User12300 commented 7 years ago

Thank you fo your help. The link was very helpful. But, I still have some questions since I didn't understand what caused the problem for my case and I didn't find the right outputs yet.

Do I have to start solr after running this command: docker run -it sparkler-local ?

When I run (following the link that you sent) : 1) docker run -it sparkler-local 2) /data/solr/bin/solr start it shows this error:

Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr_gc.log.0.current -> /data/solr/server/logs/archived/solr_gc.log.0.current
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at org.apache.solr.util.SolrCLI$UtilsTool.archiveGcLogs(SolrCLI.java:3903)
    at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3872)
    at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old GC logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr-8983-console.log -> /data/solr/server/logs/archived/solr-8983-console.log
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at org.apache.solr.util.SolrCLI$UtilsTool.archiveConsoleLogs(SolrCLI.java:3932)
    at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3875)
    at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old console logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr.log -> /data/solr/server/logs/solr.log.1
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at org.apache.solr.util.SolrCLI$UtilsTool.rotateSolrLogs(SolrCLI.java:3975)
    at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3869)
    at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed rotating old solr logs

But when I do it in this order: 1) cd to the solr directory 2) ./bin/solr start 3) docker run -it sparkler-local

solr started succesfully. However, when I continue injecting, it still doesn't work since when using this command to inject urls:java -jar /data/sparkler/sparkler-app/target/sparkler-app-0.1-SNAPSHOT.jar inject -sf seed.txt

It shows this error:

Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at edu.usc.irds.sparkler.Main$.main(Main.scala:47)
    at edu.usc.irds.sparkler.Main.main(Main.scala)
Caused by: java.nio.file.NotDirectoryException: seed.txt is not a directory
    at edu.usc.irds.sparkler.service.Injector.stackListFiles(Injector.scala:126)
    at edu.usc.irds.sparkler.service.Injector.run(Injector.scala:86)
    at edu.usc.irds.sparkler.base.CliTool$class.run(CliTool.scala:34)
    at edu.usc.irds.sparkler.service.Injector.run(Injector.scala:44)
    at edu.usc.irds.sparkler.service.Injector$.main(Injector.scala:147)
    at edu.usc.irds.sparkler.service.Injector.main(Injector.scala)
    ... 6 more

what I understood from your reply is that you have changed permissions of solr in the project. So, I tried to redownload the project and I run: mvn clean install. This was the output:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:2.1.1:war (default-war) on project sparkler-ui: Execution default-war of goal org.apache.maven.plugins:maven-war-plugin:2.1.1:war failed: basedir /home/student/work/sparkler_project/sparkler-master/sparkler-ui/banana/dist does not exist -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

I am having problems to understand the errors and what should I do to resolve that. Thank you.

buggtb commented 7 years ago

Sorry my fault, git pull and try again please.

Tom

On Mon, Aug 7, 2017 at 3:41 PM, User12300 notifications@github.com wrote:

Thank you fo your help. The link was very helpful. But, I still have some questions since I didn't understand what caused the problem for my case and I didn't find the right outputs yet.

Do I have to start solr after running this command: docker run -it sparkler-local ?

When I run (following the link that you sent) : 1) docker run -it sparkler-local 2) /data/solr/bin/solr start it shows this error:

Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr_gc.log.0.current -> /data/solr/server/logs/archived/solr_gc.log.0.current at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.solr.util.SolrCLI$UtilsTool.archiveGcLogs(SolrCLI.java:3903) at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3872) at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257) Failed archiving old GC logs Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr-8983-console.log -> /data/solr/server/logs/archived/solr-8983-console.log at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.solr.util.SolrCLI$UtilsTool.archiveConsoleLogs(SolrCLI.java:3932) at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3875) at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257) Failed archiving old console logs Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr.log -> /data/solr/server/logs/solr.log.1 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.solr.util.SolrCLI$UtilsTool.rotateSolrLogs(SolrCLI.java:3975) at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3869) at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257) Failed rotating old solr logs

But when I do it in this order: 1) cd to the solr directory 2) ./bin/solr start 3) docker run -it sparkler-local

solr started succesfully. However, when I continue injecting, it still doesn't work since when using this command to inject urls:java -jar /data/sparkler/sparkler-app/target/sparkler-app-0.1-SNAPSHOT.jar inject -sf seed.txt

It shows this error:

Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at edu.usc.irds.sparkler.Main$.main(Main.scala:47) at edu.usc.irds.sparkler.Main.main(Main.scala) Caused by: java.nio.file.NotDirectoryException: seed.txt is not a directory at edu.usc.irds.sparkler.service.Injector.stackListFiles(Injector.scala:126) at edu.usc.irds.sparkler.service.Injector.run(Injector.scala:86) at edu.usc.irds.sparkler.base.CliTool$class.run(CliTool.scala:34) at edu.usc.irds.sparkler.service.Injector.run(Injector.scala:44) at edu.usc.irds.sparkler.service.Injector$.main(Injector.scala:147) at edu.usc.irds.sparkler.service.Injector.main(Injector.scala) ... 6 more

what I understood from your reply is that you have changed permissions of solr in the project. So, I tried to redownload the project and I run: mvn clean install. This was the output:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:2.1.1:war (default-war) on project sparkler-ui: Execution default-war of goal org.apache.maven.plugins:maven-war-plugin:2.1.1:war failed: basedir /home/student/work/sparkler_project/sparkler-master/sparkler-ui/banana/dist does not exist -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

I am having problems to understand the errors and what should I do to resolve that. Thank you.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/USCDataScience/sparkler/issues/124#issuecomment-320682912, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGUeFjm2XyBfyTdJYiPkgQqHzoh603bks5sVyIogaJpZM4OuO08 .

-- Tom Barber CTO Spicule LTD tom@spicule.co.uk

http://spicule.co.uk

@spiculeim http://twitter.com/spiculeim

Schedule a meeting with me http://meetme.so/spicule

GB: +44(0)5603641316 US: +18448141689

https://leanpub.com/juju-cookbook

User12300 commented 7 years ago

Ok mvn clean install is running successfully. However, the error persists when I run: docker run -it sparkler-local then /data/solr/bin/solr start.

The error: 

sparkler@94ac27b70903:/data$ /data/solr/bin/solr start
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr_gc.log.0.current -> /data/solr/server/logs/archived/solr_gc.log.0.current
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at org.apache.solr.util.SolrCLI$UtilsTool.archiveGcLogs(SolrCLI.java:3903)
    at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3872)
    at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old GC logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr-8983-console.log -> /data/solr/server/logs/archived/solr-8983-console.log
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at org.apache.solr.util.SolrCLI$UtilsTool.archiveConsoleLogs(SolrCLI.java:3932)
    at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3875)
    at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old console logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr.log -> /data/solr/server/logs/solr.log.1
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at org.apache.solr.util.SolrCLI$UtilsTool.rotateSolrLogs(SolrCLI.java:3975)
    at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3869)
    at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed rotating old solr logs

ERROR: Logs directory /data/solr/server/logs is not writable. Exiting

Can you help me to resolve it

dmalaescu commented 7 years ago

Same error also here:

sparkler@94ac27b70903:/data$ /data/solr/bin/solr start
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr_gc.log.0.current -> /data/solr/server/logs/archived/solr_gc.log.0.current
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at org.apache.solr.util.SolrCLI$UtilsTool.archiveGcLogs(SolrCLI.java:3903)
    at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3872)
    at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old GC logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr-8983-console.log -> /data/solr/server/logs/archived/solr-8983-console.log
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at org.apache.solr.util.SolrCLI$UtilsTool.archiveConsoleLogs(SolrCLI.java:3932)
    at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3875)
    at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old console logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr.log -> /data/solr/server/logs/solr.log.1
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
    at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
    at java.nio.file.Files.move(Files.java:1395)
    at org.apache.solr.util.SolrCLI$UtilsTool.rotateSolrLogs(SolrCLI.java:3975)
    at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3869)
    at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed rotating old solr logs

ERROR: Logs directory /data/solr/server/logs is not writable. Exiting

Can you please help ? I'm running the docker container that has both solr and spakler

chrismattmann commented 7 years ago

@thammegowda do you have any idea what's up here?

thammegowda commented 7 years ago

@chrismattmann It's permission issue due to user mismatch. we were using root user earlier (inside docker), and everything was fine. However, recently the user name was switched to sparkler (reason: to take away super user privileges from sparkler processes).

That's where these permission issues appeared

@buggtb tried to fix it here by giving permission to sparkler user: https://github.com/USCDataScience/sparkler/commit/fd27abed46a948d095446e3425641f1f7be02a96 However, it is not completely fixed! @buggtb help

thammegowda commented 7 years ago

@dmalaescu

Please run bin/dockler.sh script to get bash inside sparkler's docker image. It uses root user inside docker which has all the permissions (note: root user is inside docker, it is not the same as root user in your host OS, so it can't do any harm to your host operating system)

chrismattmann commented 7 years ago

thanks @thammegowda

dmalaescu commented 6 years ago

@thammegowda Thanks, it worked. I've managed to complete the steps with root

thammegowda commented 6 years ago

@dmalaescu

I am glad it worked. FYI I fixed the issues and released a new docker image to dockerhub. Next time you want to use sparkler, just do 

docker run -it -p 8983:8983 -p 4040:4040 uscdatascience/sparkler

CC @chrismattmann

chrismattmann commented 6 years ago

thanks great work @thammegowda