Plugin fails with memory error when Disable public internet access is checked

[jrsamson]

We checked "Disable public internet access" and getting the error when we execute: "cf dfsh dataflow"

cf cli version - 6.43.0+815ea2f3d.2019-02-20 pas=2.4.2 mysql=2.5.3 rabbitmq=1.15.3 scdf version 1.3.2 vsphere

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x20 pc=0x6e6499]

goroutine 1 [running]:
github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/download/cache.fileExists(0xc042268270, 0xc6, 0x31)
        /go/src/github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/download/cache/cache.go:188 +0x89
github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/download/cache.(*fileCacheEntry).Retrieve(0xc0422662d0, 0xc042015860, 0x95, 0x935220, 0xc0422662d0, 0xc04226a032, 0xc0420737d0)
        /go/src/github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/download/cache/cache.go:132 +0x43
github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/download.(*downloader).DownloadFile(0xc042073a18, 0xc042015860, 0x95, 0x0, 0x0, 0x937f20, 0xc04203ddc0, 0x0, 0x7, 0x1, ...)
        /go/src/github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/download/download.go:152 +0x90
main.downloadAndRunShell(0x7ad5fe, 0x8, 0xc0421273e0, 0xc042073b10, 0x9335e0, 0xc042004018, 0x9374e0, 0xc0420e0320)
        /go/src/github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/plugin.go:141 +0x1b8
main.(*Plugin).Run.func3(0x9335e0, 0xc042004018, 0xc0420e82c0, 0x3d, 0x9335e0, 0xc042004018)
        /go/src/github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/plugin.go:91 +0x269
github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/format.RunAction(0x93af20, 0xc0420ee130, 0xc0420e82c0, 0x3d, 0xc042073d10, 0x9335e0, 0xc042004018, 0x7c34e8)
        /go/src/github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/format/action.go:41 +0xae
main.runAction(0xc042073cd8, 0x93af20, 0xc0420ee130, 0xc0420e82c0, 0x3d, 0xc042073d10)
        /go/src/github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/plugin.go:227 +0x90
main.(*Plugin).Run(0x982e70, 0x93af20, 0xc0420ee130, 0xc04200e460, 0x2, 0x2)
        /go/src/github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/plugin.go:92 +0x4ab
github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/vendor/code.cloudfoundry.org/cli/plugin.Start(0x935260, 0x982e70)
        /go/src/github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/vendor/code.cloudfoundry.org/cli/plugin/plugin_shim.go:34 +0x294
main.main()
        /go/src/github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/plugin.go:219 +0x6d

[csterwa]

@jrsamson Unfortunately, the SCDF for PCF plugin only works when it can access repo.spring.io to download the plugin JAR file that is linked in the Data Flow server's about API response. We should clean up the error that is occurring so that it is not as cryptic though.

A workaround is mentioned in the following issue:

https://github.com/pivotal-cf/spring-cloud-dataflow-for-pcf-cli-plugin/issues/6

It is definitely not the easiest workaround but is what we can do for now. We may need to provide a new command for adding a JAR version to local cache so that the plugin doesn't try to access public Internet. The process would then be:

1. Download public JAR for version linked in the Data Flow server's about API response
2. Run command to add it into local cache

At that point, the command should start working with your service instance. Would that work as an approach in such an environment as yours?

[csterwa]

A proposed solution for this is to add a new command on the plugin to add a downloaded jar to the CLI cache.

GIVEN a valid SCDF shell jar is downloaded into current directory WHEN cf cache-shell-jar [jarfile] THEN SCDF shell jar will be processed and added to CLI cache (including SHA generation) for use with cf dataflow-shell ... command

My one concern is that there may still be an issue for public Maven repo access from inside SCDF shell commands but that can be added as issue in that OSS repo. Would this work for you @jrsamson?

[jrsamson]

@csterwa - Yes, the proposed solution to add new command on the plugin is good.

Would it be possible to also checked access repo.spring.io. If it fails and downloaded jar does not exists in current directory - return a message to download jar and manually add to cache? And also, capability to a update the jar in cache (for newer release)?

[csterwa]

@jrsamson I like those suggestions. Since each version is kept in cache, a newer version can be imported by running the command again. We don't have a way to switch between versions and we'd rather not implement that if it is not necessary to use an older version. I can't think of a use case for that at this time.

For the no access to public Internet, I think the following would cover that:

GIVEN an existing p-dataflow service instance named mydataflow AND shell has no public Internet access (or at least not to repo.spring.io) WHEN cf dataflow-shell mydataflow is executed THEN error is posted to shell that includes message that URL was not accessible AND error message also mentions that the cf cache-shell-jar command may be used to import jar manually from URL

[jrsamson]

@csterwa That would be great! Thanks.