Error: No jar URL found for version '4.2.2' in versions.json from 'https://neo4j-contrib.github.io/neo4j-apoc-procedures/versions.json'

[survivant]

I recreate my cluster yesterday (reinstall new OS on each nodes). I'm trying to redeploy neo4j in the cluster, but I got this error

apiVersion: v1
appVersion: 4.2.2
description: Neo4j is the world's leading graph database
home: https://www.neo4j.com
icon: http://info.neo4j.com/rs/773-GON-065/images/neo4j_logo.png
keywords:
- graph
- database
- cypher
maintainers:
- email: david.allen@neo4j.com
  name: moxious
- email: bledi.feshti@neo4j.com
  name: bfeshti
- email: mark.needham@neo4j.com
  name: mneedham
name: neo4j
sources:
- https://github.com/neo4j/neo4j
- https://github.com/neo4j/docker-neo4j
- https://github.com/neo4j-contrib/neo4j-helm
version: 4.2.2-1

root@test-pcl109:~# kubectl logs twin-neo4j-core-0 --follow
Configuration override prefix = twin_neo4j_core_0
Starting Neo4j CORE 0 on twin-neo4j-core-0.twin-neo4j.default.svc.cluster.local
Warning: Some files inside "/data" are not writable from inside container. Changing folder owner to neo4j.
Changed password for user 'neo4j'.
VM settings:
    Min. Heap Size: 4.00G
    Max. Heap Size: 4.00G
    Using VM: OpenJDK 64-Bit Server VM

Fetching versions.json for Plugin 'apoc' from https://neo4j-contrib.github.io/neo4j-apoc-procedures/versions.json
Error: No jar URL found for version '4.2.2' in versions.json from 'https://neo4j-contrib.github.io/neo4j-apoc-procedures/versions.json'

root@test-pcl109:~#

in the init process, the script is trying to retrieve apoc with version 4.2.2 but in 'https://neo4j-contrib.github.io/neo4j-apoc-procedures/versions.json' the versions have 4 digits : 4.2.2.0

like

{
        "neo4j": "4.2.1",
        "neo4jVersion": "4.2.1",
        "apoc": "4.2.0.0",
        "version": "4.2.0.0",
        "url": "http://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/4.2.0.0",
        "homepageUrl": "http://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/4.2.0.0",
        "jar": "https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/download/4.2.0.0/apoc-4.2.0.0-all.jar",
        "downloadUrl": "https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/download/4.2.0.0/apoc-4.2.0.0-all.jar",
        "sha1": "e58cb627d96260c8a062692320c2f09ba66b0641",
        "sha256": "ab8dff9cbb5e4185f4242c2a111e50e0d368c28234e9e5824915f0f9842c1887",
        "md5": "c819a1a7d5b822ef2676d966e4d29d25",
        "config": {
            "+:dbms.security.procedures.unrestricted": [
                "apoc.*"
            ]
        }
    },

[eastlondoner]

@survivant 4.2.2 here refers to the neo4jVersion. The script running in the container is looking up which version of apoc it should use with its version of Neo4j.

This should work. You can test is simply with docker:

$ docker run -it --env NEO4JLABS_PLUGINS='["apoc"]' --env=NEO4J_ACCEPT_LICENSE_AGREEMENT=yes neo4j:4.2.2-enterprise 

Fetching versions.json for Plugin 'apoc' from https://neo4j-contrib.github.io/neo4j-apoc-procedures/versions.json
Installing Plugin 'apoc' from https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/download/4.2.0.1/apoc-4.2.0.1-all.jar to /var/lib/neo4j/plugins/apoc.jar 
Applying default values for plugin apoc to neo4j.conf
Directories in use:
  home:         /var/lib/neo4j
  config:       /var/lib/neo4j/conf
  logs:         /logs
  plugins:      /var/lib/neo4j/plugins
  import:       /var/lib/neo4j/import
  data:         /var/lib/neo4j/data
  certificates: /var/lib/neo4j/certificates
  run:          /var/lib/neo4j/run
Starting Neo4j.

A common cause of problems here is that the pod doesn't actually have access to the public internet and so the call to https://neo4j-contrib.github.io/neo4j-apoc-procedures/versions.json does not work.

You can stop the helm chart from trying to download the apoc plugin by setting plugins: "[]" in your values.yaml.

In production and for any environment where neo4j containers cannot fetch the versions.json from github.io we recommend that you use an initContainer that contains the plugins and links or copies them into a shared /plugins volume.

[survivant]

thanks. I though it was looking for apoc 4.2.0 but it's looking for the json of Neo4j 4.2.0 and extract the apoc version from that item.

you are right. I can't access outside my docker container

--2021-03-16 13:21:53--  https://neo4j-contrib.github.io/neo4j-apoc-procedures/versions.json
Resolving neo4j-contrib.github.io (neo4j-contrib.github.io)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘neo4j-contrib.github.io’
root@twin-neo4j-core-0:/var/lib/neo4j#

[survivant]

I'm not able to access internet from neo4j pod, but it works on other pods on the same node. There is something in the docker image that block that ?

from ingress-controler

root@test-pcl109:/home/comact# kubectl exec iep-nginx-controller-vhvrd -it -- sh
/etc/nginx $ nslookup google.com
Server:         10.1.1.191
Address:        10.1.1.191:53

Non-authoritative answer:
Name:   google.com
Address: 2607:f8b0:400b:80f::200e

Non-authoritative answer:
Name:   google.com
Address: 172.217.1.14

/etc/nginx $ wget www.google.com
Connecting to www.google.com (172.217.1.4:80)
saving to 'index.html'
index.html           100% |************************************************************************************************************************************************************************| 13319  0:00:00 ETA
'index.html' saved
/etc/nginx $

from neo4j pod

root@test-pcl109:/home/comact# kubectl exec twin-neo4j-core-0 -it -- sh
# nslookup google.com
sh: 1: nslookup: not found
# wget www.google.com
--2021-03-16 13:40:33--  http://www.google.com/
Resolving www.google.com (www.google.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘www.google.com’
#

from that docker image

Containers:
  twin-neo4j:
    Container ID:  docker://48b26692a6c101bc6842bcb2ab1526c8c9468237a7eb4d933efd47dd70e3ccf7
    Image:         neo4j:4.2.3
    Image ID:      docker-pullable://neo4j@sha256:7ac9387fd60c7fc2f2b983567dd838e181de40fb99c8c6aec6147b6699012dfb
    Ports:         5000/TCP, 7000/TCP, 6000/TCP, 7474/TCP, 7687/TCP, 7688/TCP, 3637/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP

[eastlondoner]

@survivant I'm not sure - I think this will be down to the network configuration in your k8s environment.

This definitely works in some k8s setups (we test on GKE for example and it works OK).

The method of downloading plugins from the internet at runtime is supposed to be a convenience for rapid development. But if it's not working I would urge you to turn it off plugins: "[]" and provide plugins via an initContainer instead.

n.b. in 4.2 a lot of the most used apoc functionality is included in neo4j by default (without requiring an extra apoc plugin) so it may be that you don't need this anyway.

[survivant]

thanks for your time. Now it's working. I didn't touch anything special. Probably a DNS issue on my side.

^Croot@test-pcl109:/home/comact# kubectl logs twin-neo4j-core-0 --follow
Configuration override prefix = twin_neo4j_core_0
Starting Neo4j CORE 0 on twin-neo4j-core-0.twin-neo4j.default.svc.cluster.local
Warning: Folder mounted to "/data" is not writable from inside container. Changing folder owner to neo4j.
Changed password for user 'neo4j'.
VM settings:
    Max. Heap Size (Estimated): 1.45G
    Using VM: OpenJDK 64-Bit Server VM

Fetching versions.json for Plugin 'apoc' from https://neo4j-contrib.github.io/neo4j-apoc-procedures/versions.json
Installing Plugin 'apoc' from https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/download/4.2.0.1/apoc-4.2.0.1-all.jar to /plugins/apoc.jar
Applying default values for plugin apoc to neo4j.conf
Directories in use:
  home:         /var/lib/neo4j
  config:       /var/lib/neo4j/conf
  logs:         /data/logs
  plugins:      /plugins
  import:       /var/lib/neo4j/import
  data:         /var/lib/neo4j/data
  certificates: /var/lib/neo4j/certificates
  run:          /var/lib/neo4j/run
Starting Neo4j.
VM settings:
    Max. Heap Size (Estimated): 1.45G
    Using VM: OpenJDK 64-Bit Server VM

2021-03-16 14:20:01.251+0000 WARN  Unrecognized setting. No declared setting with name: metrics.graphite.server
2021-03-16 14:20:01.257+0000 WARN  Unrecognized setting. No declared setting with name: causal_clustering.minimum_core_cluster_size_at_formation
2021-03-16 14:20:01.258+0000 WARN  Unrecognized setting. No declared setting with name: TWIN.API.SERVICE.PORT.8080.TCP.ADDR
2021-03-16 14:20:01.258+0000 WARN  Unrecognized setting. No declared setting with name: TWIN.API.SERVICE.PORT
2021-03-16 14:20:01.258+0000 WARN  Unrecognized setting. No declared setting with name: TWIN.API.SERVICE.SERVICE.HOST
2021-03-16 14:20:01.258+0000 WARN  Unrecognized setting. No declared setting with name: causal_clustering.minimum_core_cluster_size_at_runtime
2021-03-16 14:20:01.258+0000 WARN  Unrecognized setting. No declared setting with name: metrics.graphite.enabled
2021-03-16 14:20:01.258+0000 WARN  Unrecognized setting. No declared setting with name: metrics.prometheus.endpoint
2021-03-16 14:20:01.258+0000 WARN  Unrecognized setting. No declared setting with name: causal_clustering.raft_advertised_address
2021-03-16 14:20:01.259+0000 WARN  Unrecognized setting. No declared setting with name: metrics.graphite.interval
2021-03-16 14:20:01.259+0000 WARN  Unrecognized setting. No declared setting with name: causal_clustering.discovery_type
2021-03-16 14:20:01.259+0000 WARN  Unrecognized setting. No declared setting with name: TWIN.API.SERVICE.PORT.8080.TCP
2021-03-16 14:20:01.259+0000 WARN  Unrecognized setting. No declared setting with name: TWIN.API.SERVICE.PORT.8080.TCP.PROTO
2021-03-16 14:20:01.259+0000 WARN  Unrecognized setting. No declared setting with name: causal_clustering.transaction_advertised_address
2021-03-16 14:20:01.259+0000 WARN  Unrecognized setting. No declared setting with name: dbms.backup.enabled
2021-03-16 14:20:01.259+0000 WARN  Unrecognized setting. No declared setting with name: metrics.jmx.enabled
2021-03-16 14:20:01.259+0000 WARN  Unrecognized setting. No declared setting with name: causal_clustering.kubernetes.label_selector
2021-03-16 14:20:01.260+0000 WARN  Unrecognized setting. No declared setting with name: causal_clustering.kubernetes.service_port_name
2021-03-16 14:20:01.260+0000 WARN  Unrecognized setting. No declared setting with name: TWIN.API.SERVICE.SERVICE.PORT
2021-03-16 14:20:01.260+0000 WARN  Unrecognized setting. No declared setting with name: dbms.backup.listen_address
2021-03-16 14:20:01.260+0000 WARN  Unrecognized setting. No declared setting with name: causal_clustering.discovery_advertised_address
2021-03-16 14:20:01.260+0000 WARN  Unrecognized setting. No declared setting with name: TWIN.API.SERVICE.SERVICE.PORT.HTTP
2021-03-16 14:20:01.260+0000 WARN  Unrecognized setting. No declared setting with name: TWIN.API.SERVICE.PORT.8080.TCP.PORT
2021-03-16 14:20:01.260+0000 WARN  Unrecognized setting. No declared setting with name: metrics.prometheus.enabled
2021-03-16 14:20:01.261+0000 WARN  Unrecognized setting. No declared setting with name: metrics.csv.interval
2021-03-16 14:20:01.261+0000 WARN  Unrecognized setting. No declared setting with name: metrics.csv.enabled
2021-03-16 14:20:01.263+0000 INFO  Starting...
2021-03-16 14:20:07.541+0000 INFO  ======== Neo4j 4.2.3 ========

[eastlondoner]

You're welcome. Closing this issue