search

jfrog / artifactory-docker-examples

Examples for using Artifactory Docker distribution in various environments
https://www.jfrog.com/artifactory/
Apache License 2.0
330 stars 299 forks source link

Fresh installation of Artifactory OSS 7.12.6 doesn't start, missing jffe service #210

Open wl2776 opened 3 years ago

wl2776 commented 3 years ago

I have done everything according to manuals from https://jfrog.com/open-source/

My Ubuntu version is 20.04. Since there is no "focal" in https://releases.jfrog.io/artifactory/artifactory-debs/, I've added "bionic":

$ cat /etc/apt/sources.list.d/artifactory.list
deb https://releases.jfrog.io/artifactory/artifactory-debs  bionic main

Then I've installed jfrog-artifactory-oss version 7.12.6 and tried launching the service. It has launched, but browser, connected to ports :8081 or :8082 of localhost, shows that 3 services don't start:

image

Attached is the relevant part of logs:

art.log

amithins commented 3 years ago

Can you please compress and upload the content of /opt/jfrog/artifactory/var/log

wl2776 commented 3 years ago

Here it is artifactory-oss-log.tar.gz

Suphax commented 3 years ago

Having the same issue on Ubuntu 20.04, on a fresh intall

$ cat /etc/apt/sources.list.d/jfrog.list
deb https://jfrog.bintray.com/artifactory-debs bionic main

artifactory-oss-log.tar.gz

Edit: I it running by purging Artifactory, installing an older version, and starting with a different command

DON'T DO THIS IF YOU HAVE ANY CONFIG YOU WISH TO KEEP

sudo apt purge jfrog-artifactory-purge
sudo rm -rf /opt/jfrog/artifactory
sudo rm -rf /var/opt/jfrog
sudo apt update
sudo apt install jfrog-artifactory-oss=7.12.5
sudo service artifactory start

Wait for a little then try to open the page, all should be good.

amithins commented 3 years ago

@Suphax how long did you wait for services to come up in 7.12.6 ? In a VM i tried, it took me around 28 seconds for all the services to come up.

Suphax commented 3 years ago

@Suphax how long did you wait for services to come up in 7.12.6 ? In a VM i tried, it took me around 28 seconds for all the services to come up.

Roughly 30 minutes

Iril commented 3 years ago

We are also seeing this issue on a Redhat 7 New Installation:

2021-02-16T23:49:15.747Z [jffe ] [ERROR] [ ] [ ] [main ] - ArtifactoryClient::http [get] request to /api/system/ping failed with 502 code 2021-02-16T23:49:16.752Z [jffe ] [ERROR] [ ] [ ] [main ] - ArtifactoryClient::http [get] request to /api/system/ping failed with 502 code 2021-02-16T23:49:17.757Z [jffe ] [ERROR] [ ] [ ] [main ] - ArtifactoryClient::http [get] request to /api/system/ping failed with 502 code

and the status shows a Service is Healthy missing jffe error as well. Thanks Martin

RRadziejewski commented 3 years ago

2021-03-26T12:53:45.078Z [jffe ] [ERROR] [ ] [ ] [main ] - ArtifactoryClient::http [get] request to /api/system/ping failed with 404 code

saas2813 commented 3 years ago

Can anyone tell me what this JFFE service is? I have an Artifactory that suddenly stoped working and the web-page reports,: "Missing services: jffe" For some reason it got back online after restarting it but I would really like to know what it was.

grierellis commented 3 years ago

I found that libvirtd was running. It wasn't libvirtd, itself, that prevented artifactory from coming up. It was the fact that the virbr0 interface was up. It doesn't seem to be a problem bringing this interface back up after artifactory comes up.

jordaniac89 commented 2 years ago

I'm also having this issue. These three services do not start up. Seeing lots of "connection refused" errors from the different services as well as a java stacktrace when trying to initialize Home

2022-05-03T16:26:40.382Z [jfrt ] [ERROR] [ad8423ca882374a8] [tifactoryHomeConfigListener:55] [Catalina-utility-1  ] - Failed initializing Home. Caught exception: 
java.lang.IllegalStateException: Failed to initialize file watcher.
    at org.jfrog.config.watch.FileWatcher.create(FileWatcher.java:65)
    at org.jfrog.config.ConfigurationManagerImpl.<init>(ConfigurationManagerImpl.java:82)
    at org.jfrog.config.ConfigurationManagerImpl.create(ConfigurationManagerImpl.java:63)
    at org.artifactory.lifecycle.webapp.servlet.BasicConfigurationManager.<init>(BasicConfigurationManager.java:84)
    at org.artifactory.lifecycle.webapp.servlet.ArtifactoryHomeConfigListener.getBasicConfigManagers(ArtifactoryHomeConfigListener.java:78)
    at org.artifactory.lifecycle.webapp.servlet.ArtifactoryHomeConfigListener.contextInitialized(ArtifactoryHomeConfigListener.java:52)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4768)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5230)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:726)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:698)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:696)
    at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:690)
    at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1889)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Function not implemented
    at java.base/sun.nio.fs.LinuxWatchService.<init>(LinuxWatchService.java:64)
    at java.base/sun.nio.fs.LinuxFileSystem.newWatchService(LinuxFileSystem.java:47)
    at org.jfrog.config.watch.JdkWatchService.create(JdkWatchService.java:24)
    at org.jfrog.config.watch.WatchServiceAdapter.<init>(WatchServiceAdapter.java:21)
    at org.jfrog.config.watch.JdkWatchService.<init>(JdkWatchService.java:19)
    at org.jfrog.config.watch.FileWatcher.create(FileWatcher.java:63)
    ... 20 common frames omitted
2022-05-03T16:26:40.420Z [jfrt ] [ERROR] [ad8423ca882374a8] [actoryContextConfigListener:90] [Catalina-utility-1  ] - Failed initializing Artifactory context: Artifactory home not initialized.

Eventually, after many failed pings, the services all shut down.

bryaend commented 2 years ago

I've been getting stuck on this for the better part of the last day now. What I think is happening is the services are starting up before the jfrt service is fully bootstrapped. The other services seem to be smart enough (or have timeouts high enough) that they'll continue retrying until they're able to connect to the router at localhost:8046, however, the jffe (JFrog Front End) service appears to be dying out and not restarting itself, thus causing the issue. This explains why a restart fixes it.

artifactory      | 2022-09-14T14:54:26.787Z [jffe ] [INFO ] [                ] [                              ] [main                ] - pinging artifactory, attempt number 180
artifactory      | 2022-09-14T14:54:26.816Z [jffe ] [INFO ] [                ] [                              ] [main                ] - pinging artifactory attempt number 180 failed with code : ECONNREFUSED
artifactory      | 2022-09-14T14:54:27.834Z [jffe ] [ERROR] [                ] [                              ] [main                ] - Error starting application - Error: Failed pinging artifactory for 180connect ECONNREFUSED 127.0.0.1:8046
artifactory      | 2022-09-14T14:54:27.947Z [jffe ] [INFO ] [                ] [                              ] [main                ] - exit detected
artifactory      | 2022-09-14T14:54:27.948Z [jffe ] [INFO ] [                ] [                              ] [main                ] - doing clean up
artifactory      | 2022-09-14T14:54:27.949Z [jffe ] [INFO ] [                ] [                              ] [main                ] - unregistering from router
artifactory      | 2022-09-14T14:54:27.950Z [jffe ] [INFO ] [                ] [                              ] [main                ] - client is already unregistered from router
artifactory      | 2022-09-14T14:54:27.951Z [jffe ] [INFO ] [                ] [                              ] [main                ] - exit code : 0
artifactory      | 2022-09-14T14:54:27.951Z [jffe ] [INFO ] [                ] [                              ] [main                ] - closing recurring jobs
artifactory      | 2022-09-14T14:54:27.952Z [jffe ] [INFO ] [                ] [                              ] [main                ] - closing recurring jobs

The jfac service, for example, appears to keep trying to connect well past 180 seconds (3 minutes), unlike the jffe service:

artifactory      | 2022-09-14T14:56:58.186Z [jfac ] [WARN ] [24718c04392e7d4b] [o.j.c.ExecutionUtils:177      ] [jf-common-pool-1    ] - Retry 400 Elapsed 3.46 minutes failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again
artifactory      | 2022-09-14T14:57:03.200Z [jfac ] [WARN ] [24718c04392e7d4b] [o.j.c.ExecutionUtils:177      ] [jf-common-pool-1    ] - Retry 410 Elapsed 3.55 minutes failed: Registration with router on URL http://localhost:8046 failed with error: UNAVAILABLE: io exception. Trying again

I'll see if I can gather enough evidence to open a bug with JFrog for this. I had to do the same workaround and start with 7.12.6 and then change up the version which is not how any of this should work.

eldada commented 2 years ago

This is an old issue with an old version. I recommend you use the latest version of the Artifactory image as there are many improvements there since 7.12. Latest version is 7.41.12.

bryaend commented 2 years ago

This is an old issue with an old version. I recommend you use the latest version of the Artifactory image as there are many improvements there since 7.12. Latest version is 7.41.12.

Please re-read :) The only way to get this to work is to start with 7.12.6 and then upgrade after it is bootstrapped.

bryaend commented 2 years ago

logs.txt FWIW this exact scenario is still happening with 7.41.12 on a fresh install. Logs attached. This was run per the documentation at https://www.jfrog.com/confluence/display/JFROG/Installing+Artifactory#InstallingArtifactory-DockerInstallation

eldada commented 2 years ago

Thx. I'll have someone in the team review this again with the newer version.

mhvelplund commented 2 years ago

I ran into this issue while trying to bootstrap the configuration of my server. I installed the latest version (7.41.12?) on an Amazon Linux 2 instance in AWS and it started fine.

Then I added users and repositories, and copied the config- and security descriptors from the admin page to /var/opt/jfrog/artifactory/etc/artifactory as artifactory.config.import.xml and security.import.xml respectively. Finally I added a artifactory.lic with the license and a binarystore.xml.

After rebooting, I hade the problem described above. By tinkering with the configuration over multiple resets, I found out that the jffe service would fail to start if any user in the security import file had encoded <userProperty> elements.

Funnily enough, the admin user can have an unencoded user property (the one it has after the first boot with JSON), but after completing the setup, the security descriptor in the export contains some sort of encoded or encrypted value, and that will cause the server to fail on start-up, hanging forever while waiting for the jffe service.

I wasn't able to find anything in the log files about invalid configurations though, and it was purely (protracted) trial and error that led me to this.

bryaend commented 2 years ago

I ran into this issue while trying to bootstrap the configuration of my server. I installed the latest version (7.41.12?) on an Amazon Linux 2 instance in AWS and it started fine.

Then I added users and repositories, and copied the config- and security descriptors from the admin page to /var/opt/jfrog/artifactory/etc/artifactory as artifactory.config.import.xml and security.import.xml respectively. Finally I added a artifactory.lic with the license and a binarystore.xml.

After rebooting, I hade the problem described above. By tinkering with the configuration over multiple resets, I found out that the jffe service would fail to start if any user in the security import file had encoded <userProperty> elements.

Funnily enough, the admin user can have an unencoded user property (the one it has after the first boot with JSON), but after completing the setup, the security descriptor in the export contains some sort of encoded or encrypted value, and that will cause the server to fail on start-up, hanging forever while waiting for the jffe service.

I wasn't able to find anything in the log files about invalid configurations though, and it was purely (protracted) trial and error that led me to this.

I actually found something similar. I got a case open with JFrog and worked with them live to troubleshoot. They had me use their config.sh script that is available from the Downloads page for Docker Compose and it worked first try. When comparing the output of their script (which I don't like for various unrelated reasons), I found that the config it bootstrapped was far slimmer than their sample configs. I essentially took that config, stripped out the stuff I didn't want (again, for various unrelated reasons), and it works every time, now. I didn't dig enough to prove exactly what part(s) of the default sample config are to blame, but, it sounds like a similar issue to what you found. Whatever is happening, there's not enough logging to find what the error is before services start dying. This would be a good challenge for somebody to meticulously add/remove each portion of the config until they find what is causing the issues and get a bug open with JFrog.

mhvelplund commented 1 year ago

They had me use their config.sh script that is available from the Downloads page for Docker Compose and it worked first try.

Maybe add a deep link to help those who come here later?

bryaend commented 1 year ago

They had me use their config.sh script that is available from the Downloads page for Docker Compose and it worked first try.

Maybe add a deep link to help those who come here later?

It's the same as the docker-compose download link on the Download page but this looks like it should be a fairly good permalink.

https://releases.jfrog.io/artifactory/artifactory-pro/org/artifactory/pro/docker/jfrog-artifactory-pro/[RELEASE]/jfrog-artifactory-pro-[RELEASE]-compose.tar.gz

romain-cros commented 1 year ago

I had the same issue that was due to another service using the artifactory default service port 8081. To check if a process is listening on the port, the following command can be used.

netstat -tunapl | grep 8081

I fixed the issue by defining an alternative port in the system.yaml file. 

artifactory:
    port: <another port>
taxi333 commented 1 year ago

I am having the same issue on a fresh install

dotkas commented 1 year ago

I had the same issue, only to discover I was running on a M1 mac with the DOCKER_DEFAULT_PLATFORM set to amd64, causing it to run emulated. Not sure why, but running it natively on arm64 fixed it for me.

LucaIcaro commented 5 months ago

same issue on a fresh install with the latest version on Amazon Linux 2.

bmcnally-uw commented 4 months ago

I'm seeing what I think is the same behavior on Ubuntu 22.04 with jfrog-artifactory-oss version 7.84.17. console.log reports:

2024-07-12T20:00:51.079Z [jfrou] [WARN ] [29ea57953f593bb5] [local_topology_helper.go:68   ] [main                ] [] - Missing required services: [jffe]
2024-07-12T20:00:51.080Z [jfac ] [INFO ] [c35a3fcec7d78535] [o.j.c.ExecutionUtils:280      ] [jf-common-pool-0    ] - Retry 252 Elapsed 4.16 minutes failed: Router readiness on URL http://localhost:8046/router/api/v1/system/readiness failed with error: Router readiness failed with status 503. Trying again

access-service.log reports:

2024-07-12T20:05:38.005Z [jfac ] [INFO ] [c35a3fcec7d78535] [o.j.c.ExecutionUtils:280      ] [jf-common-pool-0    ] - Retry 538 Elapsed 8.94 minutes failed: Router readiness on URL http://localhost:8046/router/api/v1/system/readiness failed with error: Router readiness failed with status 503. Trying again
2024-07-12T20:05:39.008Z [jfac ] [INFO ] [c35a3fcec7d78535] [o.j.c.ExecutionUtils:280      ] [jf-common-pool-1    ] - Retry 539 Elapsed 8.96 minutes failed: Router readiness on URL http://localhost:8046/router/api/v1/system/readiness failed with error: Router readiness failed with status 503. Trying again
isaac84 commented 4 months ago

An FYI for anyone running into this issue.

I ran into this same error today on an old version of Amazon Linux. The version of node that comes bundled with Artifactory requires a newer version of GLIB_C then can be installed on this version of AL. This stops the frontend service from starting leading to the jffe service not being available. 

~]$ sudo -u artifactory DEBUG=true /opt/jfrog/artifactory/app/third-party/node/bin/node -v
/opt/jfrog/artifactory/app/third-party/node/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /opt/jfrog/artifactory/app/third-party/node/bin/node)
/opt/jfrog/artifactory/app/third-party/node/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /opt/jfrog/artifactory/app/third-party/node/bin/node)
/opt/jfrog/artifactory/app/third-party/node/bin/node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /opt/jfrog/artifactory/app/third-party/node/bin/node)
harryh-justeat commented 3 months ago

@isaac84 Thanks for this, do you know what version of node it now needs?

mooh-pooh commented 2 weeks ago

Fixed this issue by setting environment variable no_proxy=localhost,127.0.0.1