[WIP] CI update for R interpreter

bzz commented 8 years ago

Adding pre-requests for CI to pass over the -P r

[ ] figure out why unit-tests are failing

from https://s3.amazonaws.com/archive.travis-ci.org/jobs/111961332/log.txt

09:47:56,280  INFO org.apache.zeppelin.rest.NotebookRestApi:322 - move paragraph 2BEJWMVH7 20160226-094756_336283760 10
09:47:56,280 ERROR org.apache.zeppelin.rest.NotebookRestApi:341 - Exception in NotebookRestApi while moveParagraph 
java.lang.IndexOutOfBoundsException: paragraph size is 2 , index is 10
at org.apache.zeppelin.notebook.Note.moveParagraph(Note.java:271)
at org.apache.zeppelin.rest.NotebookRestApi.moveParagraph(NotebookRestApi.java:335)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)

[ ] clean-up install scripts
[ ] update it, after profile r is added

bzz commented 8 years ago

This will allow apache/incubator-zeppelin#208 to pass

elbamos commented 8 years ago

@bzz Pls let me know when I should pull this in. Is this actually working on CI? It doesn't pull-in the R dependencies that are involved in handling certain plots, and I'm pretty sure I have unit tests for those routines, so I'd think it would fail those tests.

bzz commented 8 years ago

@elbamos sure, let's merge this as soon as all todo items here are completed.

Regarding the R dependencies - I just add what you have identified as "required" in comments of ZEPPELIN-700. Or did I miss something else?

Knitr - Not required Repr - not required Evaluate - Required Base64enc - not required

By required, it usually means things, that are needed to build and pass all tests. Please let me know (better in comments to the issue above) if something else is required.

The CI status for this PR is available under r-interpreter-ci branch in https://travis-ci.org/bzz/incubator-zeppelin/branches

elbamos commented 8 years ago

Those are optional runtime dependencies. But they are test time dependencies. For example, we need the dependencies used to display images so we can test the code for handling images.

On Mar 2, 2016, at 10:42 PM, Alexander notifications@github.com wrote:

@elbamos sure, let's merge this as soon as all todo items here are completed.

Regarding the R dependencies - I just add what you have identified as "required" in comments of ZEPPELIN-700. Or did I miss something else?

Knitr - Not required Repr - not required Evaluate - Required Base64enc - not required

By required, it usually means things, that are needed to build and pass all tests. Please let me know (better in comments to the issue above) if something else is required.

— Reply to this email directly or view it on GitHub.

bzz commented 8 years ago

Well, I guess what I'm talking about is a list of all the dependencies, that are not part of the automated build process (except maven, java and R) that are needed to be present on the machine in order to build and pass test for your contribution.

Can you please add such list either in original PR or in comments to the ZEPPELIN-700 (so we keep community in the loop)?

Thanks!

elbamos commented 8 years ago

I think the list you made should do it. I think the list is in the PR, and the jira discussion as well.

Note that the easiest way to install repr is with the devtools package. So ordinarily I'd say to add that.

On Mar 2, 2016, at 10:59 PM, Alexander notifications@github.com wrote:

Well, I guess what I'm talking about is a list of all the dependencies, that are not part of the automated build process (except maven and java) that are needed to be present on the machine in order to build and pass test for your contribution.

Can you please add such list either in original PR or in comments to the ZEPPELIN-700 (so we keep community in the loop)?

Thanks!

— Reply to this email directly or view it on GitHub.

bzz commented 8 years ago

Sorry, I'm confused and not sure I understand you.

Could you please post such a list, where things that needed to build and pass tests over your contribution are listed explicitly?

Thanks.

elbamos commented 8 years ago

Build-Time Dependencies:

R Test-Time Dependencies:
R
Evaluate
base64enc
repr
knitr Run-Time Dependencies:
R
Evaluate

Or, if you look in the 0.5.6 branch here, I'm pretty sure the travis script does a fine job at getting the dependencies loaded. The CI problems with the PR have never been with the R dependencies.

bzz commented 8 years ago

Triggering CI

bzz commented 8 years ago

CI failures https://travis-ci.org/bzz/incubator-zeppelin/builds/118443123

RAT check fails on files \wo Apache headers in apache/incubator-zeppelin#208

2 Unknown Licenses

*******************************

Unapproved licenses:

r/src/main/scala/scala/Console.scala
r/src/main/scala/org/apache/zeppelin/rinterpreter/rscala/Package.scala

*******************************

UnitTest failure in ZeppelinServer

10:08:53,421 ERROR org.apache.zeppelin.rest.NotebookRestApi:341 - Exception in NotebookRestApi while moveParagraph 
java.lang.IndexOutOfBoundsException: paragraph size is 2 , index is 10
at org.apache.zeppelin.notebook.Note.moveParagraph(Note.java:271)
at org.apache.zeppelin.rest.NotebookRestApi.moveParagraph(NotebookRestApi.java:335)

Results :

Failed tests: 
ZeppelinSparkClusterTest.pySparkDepLoaderTest:167->getSparkVersionNumber:220 expected:<FINISHED> but was:<ERROR>
ZeppelinSparkClusterTest.pySparkAutoConvertOptionTest:112->getSparkVersionNumber:220 expected:<FINISHED> but was:<ERROR>
ZeppelinSparkClusterTest.basicRDDTransformationAndActionTest:81 expected:<FINISHED> but was:<ERROR>
ZeppelinSparkClusterTest.pySparkTest:90->getSparkVersionNumber:220 expected:<FINISHED> but was:<ERROR>
ZeppelinSparkClusterTest.zRunTest:152 expected:<FINISHED> but was:<ERROR>

Tests run: 35, Failures: 5, Errors: 0, Skipped: 0

elbamos commented 8 years ago

@bzz The CI errors there clearly doesn't relate to any code touched by this PR.

In now I think 8 weeks, you've asked me four times to list the PR's dependencies (which was part of CI I already had working), and pasted a CI error caused by another part of zeppelin.

Do we need to go back to the community to clarify the current consensus?

bzz commented 8 years ago

Using it as a testbed for ZEPPELIN-776

bzz commented 8 years ago

Build is failing due to SUDO usage

bzz commented 8 years ago

devtools::install_github('IRkernel/repr') is very un-relieable way to install the packages - it fails our build if GH API rate-limit is maxed out by TravisCI, same as ramnathv/rMaps#82

@elbamos is there another way to do it?

devtools::install_github('IRkernel/repr')
Downloading GitHub repo IRkernel/repr@master
from URL https://api.github.com/repos/IRkernel/repr/zipball/master
Error in download(dest, src, auth) : Forbidden (HTTP 403).

{
  "message": "API rate limit exceeded for ***.***.***.***. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
  "documentation_url": "https://developer.github.com/v3/#rate-limiting"
}

elbamos / Zeppelin-With-R

[WIP] CI update for R interpreter #12