Closed User12300 closed 6 years ago
I've not run the docker container, but basically you're not providing a path to the jar so it can't find it, either provide the path or I suspect sparkler.sh does the job for you.
java -jar /data/sparkler/...... I guess is what you're after.
Tom
On Fri, Aug 4, 2017 at 10:58 PM, User12300 notifications@github.com wrote:
I am new to Sparkler. I begun with the requirements provided and everything worked well. Then, when I started to inject some urls, I got an issue.
Here are the steps that I followed to inject urls:
- cd to the root directory of the project
- docker run -it sparkler-local
- /data/solr/bin/solr
- /data/sparkler/bin/sparkler.sh
- java -jar sparkler-app-0.1-SNAPSHOT.jar inject -sf seed.txt
The output that it must appear is:
2016-06-07 19:22:49 INFO Injector$:70 [main] - Injecting 2 seeds
jobId = sparkler-job-1465352569649
But, I got this output: Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar
I don't understand what caused this problem and how can I resolve it. I will be very gratefull if I get an answer. I look forward to your reply.
Thank you.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/USCDataScience/sparkler/issues/124, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGUeG0IrrlWm_X2B4tabPpv9UkQ87RBks5sU5QLgaJpZM4OuO08 .
-- Tom Barber CTO Spicule LTD tom@spicule.co.uk
@spiculeim http://twitter.com/spiculeim
Schedule a meeting with me http://meetme.so/spicule
GB: +44(0)5603641316 US: +18448141689
Thank you for your reply. I tried to add the path to the jar but it still doesn't work unfortunately.
When I use sparkler.sh it shows this output:
sparkler.sh: command not found
So, I tried this: bash sparkler.sh
and the output was:
Sub Commands:
inject : edu.usc.irds.sparkler.service.Injector
- Inject (seed) URLS to crawldb
crawl : edu.usc.irds.sparkler.pipeline.Crawler
- Run crawl pipeline for several iterations
And when I continue to inject, it shows the same error:
Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar
java -jar /data/sparkler/sparkler-app/target/sparkler-app-0.1-SNAPSHOT.jar
is what you're after. Although it seems to me like solr installed with the wrong permissions currently when i built it, I'll fix that.
Tom
On Sat, Aug 5, 2017 at 6:09 PM, User12300 notifications@github.com wrote:
Thank you for your reply. I tried to add the path to the jar but it still doesn't work unfortunately. When I use sparkler.sh it shows this output: sparkler.sh: command not found
So, I tried this: bash sparkler.sh and the output was:
Sub Commands: inject : edu.usc.irds.sparkler.service.Injector
- Inject (seed) URLS to crawldb crawl : edu.usc.irds.sparkler.pipeline.Crawler
- Run crawl pipeline for several iterations
And when I continue to inject, it shows the same error: Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/USCDataScience/sparkler/issues/124#issuecomment-320455398, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGUeJzJxEj4x6AkzFSx-1Z0qiEsYUkxks5sVKHVgaJpZM4OuO08 .
-- Tom Barber CTO Spicule LTD tom@spicule.co.uk
@spiculeim http://twitter.com/spiculeim
Schedule a meeting with me http://meetme.so/spicule
GB: +44(0)5603641316 US: +18448141689
As you'll see here it works now: https://asciinema.org/a/2tslGQY2MGyaGlVy8rEU0fohC
The only change I have to make is cd to /data/sparkler before running the crawl.
Tom
On Sun, Aug 6, 2017 at 3:32 AM, Tom Barber tom@spicule.co.uk wrote:
java -jar /data/sparkler/sparkler-app/target/sparkler-app-0.1- SNAPSHOT.jar
is what you're after. Although it seems to me like solr installed with the wrong permissions currently when i built it, I'll fix that.
Tom
On Sat, Aug 5, 2017 at 6:09 PM, User12300 notifications@github.com wrote:
Thank you for your reply. I tried to add the path to the jar but it still doesn't work unfortunately. When I use sparkler.sh it shows this output: sparkler.sh: command not found
So, I tried this: bash sparkler.sh and the output was:
Sub Commands: inject : edu.usc.irds.sparkler.service.Injector
- Inject (seed) URLS to crawldb crawl : edu.usc.irds.sparkler.pipeline.Crawler
- Run crawl pipeline for several iterations
And when I continue to inject, it shows the same error: Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/USCDataScience/sparkler/issues/124#issuecomment-320455398, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGUeJzJxEj4x6AkzFSx-1Z0qiEsYUkxks5sVKHVgaJpZM4OuO08 .
-- Tom Barber CTO Spicule LTD tom@spicule.co.uk
@spiculeim http://twitter.com/spiculeim
Schedule a meeting with me http://meetme.so/spicule
GB: +44(0)5603641316 <+44%2056%200364%201316> US: +18448141689 <(844)%20814-1689>
-- Tom Barber CTO Spicule LTD tom@spicule.co.uk
@spiculeim http://twitter.com/spiculeim
Schedule a meeting with me http://meetme.so/spicule
GB: +44(0)5603641316 US: +18448141689
Thank you fo your help. The link was very helpful. But, I still have some questions since I didn't understand what caused the problem for my case and I didn't find the right outputs yet.
Do I have to start solr after running this command: docker run -it sparkler-local ?
When I run (following the link that you sent) : 1) docker run -it sparkler-local
2) /data/solr/bin/solr start
it shows this error:
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr_gc.log.0.current -> /data/solr/server/logs/archived/solr_gc.log.0.current
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.solr.util.SolrCLI$UtilsTool.archiveGcLogs(SolrCLI.java:3903)
at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3872)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old GC logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr-8983-console.log -> /data/solr/server/logs/archived/solr-8983-console.log
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.solr.util.SolrCLI$UtilsTool.archiveConsoleLogs(SolrCLI.java:3932)
at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3875)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old console logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr.log -> /data/solr/server/logs/solr.log.1
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.solr.util.SolrCLI$UtilsTool.rotateSolrLogs(SolrCLI.java:3975)
at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3869)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed rotating old solr logs
But when I do it in this order: 1) cd to the solr directory
2) ./bin/solr start
3) docker run -it sparkler-local
solr started succesfully. However, when I continue injecting, it still doesn't work since when using this command to inject urls:java -jar /data/sparkler/sparkler-app/target/sparkler-app-0.1-SNAPSHOT.jar inject -sf seed.txt
It shows this error:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at edu.usc.irds.sparkler.Main$.main(Main.scala:47)
at edu.usc.irds.sparkler.Main.main(Main.scala)
Caused by: java.nio.file.NotDirectoryException: seed.txt is not a directory
at edu.usc.irds.sparkler.service.Injector.stackListFiles(Injector.scala:126)
at edu.usc.irds.sparkler.service.Injector.run(Injector.scala:86)
at edu.usc.irds.sparkler.base.CliTool$class.run(CliTool.scala:34)
at edu.usc.irds.sparkler.service.Injector.run(Injector.scala:44)
at edu.usc.irds.sparkler.service.Injector$.main(Injector.scala:147)
at edu.usc.irds.sparkler.service.Injector.main(Injector.scala)
... 6 more
what I understood from your reply is that you have changed permissions of solr in the project. So, I tried to redownload the project and I run: mvn clean install
. This was the output:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:2.1.1:war (default-war) on project sparkler-ui: Execution default-war of goal org.apache.maven.plugins:maven-war-plugin:2.1.1:war failed: basedir /home/student/work/sparkler_project/sparkler-master/sparkler-ui/banana/dist does not exist -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
I am having problems to understand the errors and what should I do to resolve that. Thank you.
Sorry my fault, git pull and try again please.
Tom
On Mon, Aug 7, 2017 at 3:41 PM, User12300 notifications@github.com wrote:
Thank you fo your help. The link was very helpful. But, I still have some questions since I didn't understand what caused the problem for my case and I didn't find the right outputs yet.
Do I have to start solr after running this command: docker run -it sparkler-local ?
When I run (following the link that you sent) : 1) docker run -it sparkler-local 2) /data/solr/bin/solr start it shows this error:
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr_gc.log.0.current -> /data/solr/server/logs/archived/solr_gc.log.0.current at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.solr.util.SolrCLI$UtilsTool.archiveGcLogs(SolrCLI.java:3903) at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3872) at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257) Failed archiving old GC logs Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr-8983-console.log -> /data/solr/server/logs/archived/solr-8983-console.log at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.solr.util.SolrCLI$UtilsTool.archiveConsoleLogs(SolrCLI.java:3932) at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3875) at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257) Failed archiving old console logs Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr.log -> /data/solr/server/logs/solr.log.1 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.solr.util.SolrCLI$UtilsTool.rotateSolrLogs(SolrCLI.java:3975) at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3869) at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257) Failed rotating old solr logs
But when I do it in this order: 1) cd to the solr directory 2) ./bin/solr start 3) docker run -it sparkler-local
solr started succesfully. However, when I continue injecting, it still doesn't work since when using this command to inject urls:java -jar /data/sparkler/sparkler-app/target/sparkler-app-0.1-SNAPSHOT.jar inject -sf seed.txt
It shows this error:
Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at edu.usc.irds.sparkler.Main$.main(Main.scala:47) at edu.usc.irds.sparkler.Main.main(Main.scala) Caused by: java.nio.file.NotDirectoryException: seed.txt is not a directory at edu.usc.irds.sparkler.service.Injector.stackListFiles(Injector.scala:126) at edu.usc.irds.sparkler.service.Injector.run(Injector.scala:86) at edu.usc.irds.sparkler.base.CliTool$class.run(CliTool.scala:34) at edu.usc.irds.sparkler.service.Injector.run(Injector.scala:44) at edu.usc.irds.sparkler.service.Injector$.main(Injector.scala:147) at edu.usc.irds.sparkler.service.Injector.main(Injector.scala) ... 6 more
what I understood from your reply is that you have changed permissions of solr in the project. So, I tried to redownload the project and I run: mvn clean install. This was the output:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:2.1.1:war (default-war) on project sparkler-ui: Execution default-war of goal org.apache.maven.plugins:maven-war-plugin:2.1.1:war failed: basedir /home/student/work/sparkler_project/sparkler-master/sparkler-ui/banana/dist does not exist -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
I am having problems to understand the errors and what should I do to resolve that. Thank you.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/USCDataScience/sparkler/issues/124#issuecomment-320682912, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGUeFjm2XyBfyTdJYiPkgQqHzoh603bks5sVyIogaJpZM4OuO08 .
-- Tom Barber CTO Spicule LTD tom@spicule.co.uk
@spiculeim http://twitter.com/spiculeim
Schedule a meeting with me http://meetme.so/spicule
GB: +44(0)5603641316 US: +18448141689
Ok mvn clean install
is running successfully. However, the error persists when I run: docker run -it sparkler-local
then /data/solr/bin/solr start
.
The error:
sparkler@94ac27b70903:/data$ /data/solr/bin/solr start
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr_gc.log.0.current -> /data/solr/server/logs/archived/solr_gc.log.0.current
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.solr.util.SolrCLI$UtilsTool.archiveGcLogs(SolrCLI.java:3903)
at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3872)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old GC logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr-8983-console.log -> /data/solr/server/logs/archived/solr-8983-console.log
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.solr.util.SolrCLI$UtilsTool.archiveConsoleLogs(SolrCLI.java:3932)
at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3875)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old console logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr.log -> /data/solr/server/logs/solr.log.1
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.solr.util.SolrCLI$UtilsTool.rotateSolrLogs(SolrCLI.java:3975)
at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3869)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed rotating old solr logs
ERROR: Logs directory /data/solr/server/logs is not writable. Exiting
Can you help me to resolve it
Same error also here:
sparkler@94ac27b70903:/data$ /data/solr/bin/solr start
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr_gc.log.0.current -> /data/solr/server/logs/archived/solr_gc.log.0.current
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.solr.util.SolrCLI$UtilsTool.archiveGcLogs(SolrCLI.java:3903)
at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3872)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old GC logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr-8983-console.log -> /data/solr/server/logs/archived/solr-8983-console.log
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.solr.util.SolrCLI$UtilsTool.archiveConsoleLogs(SolrCLI.java:3932)
at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3875)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old console logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr.log -> /data/solr/server/logs/solr.log.1
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.solr.util.SolrCLI$UtilsTool.rotateSolrLogs(SolrCLI.java:3975)
at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3869)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed rotating old solr logs
ERROR: Logs directory /data/solr/server/logs is not writable. Exiting
Can you please help ? I'm running the docker container that has both solr and spakler
@thammegowda do you have any idea what's up here?
@chrismattmann
It's permission issue due to user mismatch.
we were using root
user earlier (inside docker), and everything was fine.
However, recently the user name was switched to sparkler
(reason: to take away super user privileges from sparkler processes).
That's where these permission issues appeared
@buggtb tried to fix it here by giving permission to sparkler
user: https://github.com/USCDataScience/sparkler/commit/fd27abed46a948d095446e3425641f1f7be02a96
However, it is not completely fixed!
@buggtb help
@dmalaescu
Please run bin/dockler.sh
script to get bash inside sparkler's docker image.
It uses root user inside docker which has all the permissions (note: root user is inside docker, it is not the same as root user in your host OS, so it can't do any harm to your host operating system)
thanks @thammegowda
@thammegowda Thanks, it worked. I've managed to complete the steps with root
@dmalaescu
I am glad it worked. FYI I fixed the issues and released a new docker image to dockerhub. Next time you want to use sparkler, just do
docker run -it -p 8983:8983 -p 4040:4040 uscdatascience/sparkler
CC @chrismattmann
thanks great work @thammegowda
I am new to Sparkler. I begun with the requirements provided and everything worked well. Then, when I started to inject some urls, I got an issue.
Here are the steps that I followed to inject urls:
1) cd to the root directory of the project 2) docker run -it sparkler-local 3) /data/solr/bin/solr 4) /data/sparkler/bin/sparkler.sh 5) java -jar sparkler-app-0.1-SNAPSHOT.jar inject -sf seed.txt
The output that it must appear is:
But, I got this output:
Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar
I don't understand what caused this problem and how can I resolve it. I will be very gratefull if I get an answer. I look forward to your reply.
Thank you.