Configure Dataflow Server to use custom maven location

[ashwgupt]

I am working with Dataflow Server Local and wish to use my locally hosted URIs for registering a module or app. Based upon suggestions from other issue, I've used the file:// descriptor to register my apps with local jar files from Dataflow Shell.

However when I create the stream, the data flow server still goes out to the external repo (which is not desired due to our proxy requirements) and that cause it to fail with below exception at dataflow server shell logs: Command failed org.springframework.cloud.dataflow.rest.client.DataFlowClientException: failed to resolve MavenResource:... and at Dataflow server level with: Caused by: java.net.UnknownHostException: repo.spring.io

Is there a way to start the Dataflow server with configuration such that it uses a custom Maven repository location e.g. local or our enterprise maven repo, instead of its default one?

Apologies if this is already documented but I failed to find so.

FYI: The command used are:

> app register --name task-processor --type processor --uri file:///Users/ABC/apache-maven-3.3.9/.m2/repository/io/spring/cloud/taskprocessor/1.1.0.BUILD-SNAPSHOT/taskprocessor-1.1.0.BUILD-SNAPSHOT.jar
> app register --name task-launcher-local --type sink --uri file:///Users/ABC/apache-maven-3.3.9/.m2/repository/org/springframework/cloud/stream/app/spring-cloud-starter-stream-sink-task-launcher-local/1.1.0.BUILD-SNAPSHOT/spring-cloud-starter-stream-sink-task-launcher-local-1.1.0.BUILD-SNAPSHOT.jar
> stream create foo --definition "rabbit --rabbit.queues=cloud-stream-source --rabbit.requeue=true --spring.rabbitmq.host=localhost --spring.rabbitmq.port=5672 --spring.cloud.stream.bindings.output.contentType='text/plain' | task-processor --uri=file:///Users/ABC/apache-maven-3.3.9/.m2/repository/com/example/thumbnail-generator/0.0.1-SNAPSHOT/thumbnail-generator-0.0.1-SNAPSHOT.jar | task-launcher-local" --deploy

[ilayaperumalg]

Hi @ashwgupt,

You can install your apps locally into local maven repository. The local deployer would resolve the maven resource from there. You need to use maven resource URI (ex: maven://com.mypackage:myapp:1.0.0.BUILD-SNAPSHOT). You can find more info here

[ashwgupt]

Hi @ilayaperumalg

Unfortunately that doesn't work. If I use maven uri, data flow server attempts to download even the task processor and task launcher from external repo.

After many trials and search I could solve those by using file uri in my registration command. However I don have any control on data flow server downloading rabbit dependencies and that is where it still goes to remote Repo.

Any inputs?

[ilayaperumalg]

Unfortunately that doesn't work. If I use maven uri, data flow server attempts to download even the task processor and task launcher from external repo.

The local repo uses the https://repo.spring.io/libs-snapshot maven repo by default. But if you don't want to connect to spring repo, you can still use your enterprise maven repo by setting this property:

maven:
  remoteRepositories:
    myRepo:
      url: https://my-custom-repo

The other way around is to download the apps is via docker but it is not yet supported in local deployer: https://github.com/spring-cloud/spring-cloud-dataflow/issues/695

[ashwgupt]

Great! this solved my problem and helped me use our enterprise repository.

Thanks

[rajaramkushwaha]

We can point local maven repository by setting the file based URL also: Below is complete example: java -jar spring-cloud-dataflow-server-local-1.3.0.BUILD-SNAPSHOT.jar --maven.localRepository=mylocal --maven.remote-repositories.repo1.url=file:///C:/Users/#yourusername#/.m2/repository/

Adjust maven repo path as per your directory structure

[sidhu24]

Hi, I am getting the below exception Command failed org.springframework.cloud.dataflow.rest.client.DataFlowClientException: Failed to resolve MavenResource: com.javainuse:sink:jar:0.0.1-SNAPSHOT. Configured remote repository: : [springRepo]

[sidhu24]

how to point local maven repository ? can u please provide me an example

[hendisantika]

Unfortunately that doesn't work. If I use maven uri, data flow server attempts to download even the task processor and task launcher from external repo.

The local repo uses the https://repo.spring.io/libs-snapshot maven repo by default. But if you don't want to connect to spring repo, you can still use your enterprise maven repo by setting this property:

maven:
  remoteRepositories:
    myRepo:
      url: https://my-custom-repo

The other way around is to download the apps is via docker but it is not yet supported in local deployer: #695

Where we can setting this config @ilayaperumalg ? Thanks

[ilayaperumalg]

@hendisantika You can set this as server level property and doc for this is available here

[MariusKl]

Unfortunately that doesn't work. If I use maven uri, data flow server attempts to download even the task processor and task launcher from external repo.

The local repo uses the https://repo.spring.io/libs-snapshot maven repo by default. But if you don't want to connect to spring repo, you can still use your enterprise maven repo by setting this property:

maven:
  remoteRepositories:
    myRepo:
      url: https://my-custom-repo

The other way around is to download the apps is via docker but it is not yet supported in local deployer: #695

Hi @ilayaperumalg,

you wrote that an own repo will replace the spring repo. I added this

maven:
  remoteRepositories:
    myRepo:
      url: xxx
      auth:
        username: xxx
        password: xxx

but end up with two remote repositores, both "myRepo" and "springRepo" in MavenProperties.setRemoteRepositories. I could not find any option to remove the springRepo. Is it possible?