Can't start domain after upgrade to 3.1-36

[glassfishrobot]

I updated glassfish using updatetool. But after update domain doesn't start.

Output:

asadmin --secure start-domain --domaindir /data/glassfish/domains/ prod Waiting for prod to start ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ No response from the Domain Administration Server (prod) after 600 seconds. The command is either taking too long to complete or the server has failed. Please see the server log files for command status. Please start with the --verbose option in order to see early messages. Command start-domain failed.

asadmin --secure start-domain --verbose --domaindir /data/glassfish/domains/ prod 06.01.2011 20:18:58 com.sun.enterprise.admin.launcher.GFLauncherLogger info INFO: JVM invocation command line: /usr/java/jdk1.6.0_23/bin/java -cp /usr/local/glassfishv3/glassfish/modules/glassfish.jar -XX:+UnlockDiagnosticVMOptions -XX:MaxPermSize=192m -XX:NewRatio=2 -XX:+LogVMOutput -XX:LogFile=/data/glassfish/domains/prod/logs/jvm.log -Xmx512m -javaagent:/usr/local/glassfishv3/glassfish/lib/monitor/btrace-agent.jar=unsafe=true,noServer=true -client -Dosgi.shell.telnet.maxconn=1 -Djdbc.drivers=org.apache.derby.jdbc.ClientDriver -Dfelix.fileinstall.dir=/usr/local/glassfishv3/glassfish/modules/autostart/ -Djavax.net.ssl.keyStore=/data/glassfish/domains/prod/config/keystore.jks -Dosgi.shell.telnet.port=6666 -Djava.security.policy=/data/glassfish/domains/prod/config/server.policy -Dfelix.fileinstall.poll=5000 -Dcom.sun.aas.instanceRoot=/data/glassfish/domains/prod -Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory -Dosgi.shell.telnet.ip=127.0.0.1 -Djava.endorsed.dirs=/usr/local/glassfishv3/glassfish/modules/endorsed:/usr/local/glassfishv3/glassfish/lib/endorsed -Dcom.sun.aas.installRoot=/usr/local/glassfishv3/glassfish -Djava.ext.dirs=/usr/java/jdk1.6.0_23/lib/ext:/usr/java/jdk1.6.0_23/jre/lib/ext:/data/glassfish/domains/prod/lib/ext -Dfelix.fileinstall.bundles.new.start=true -Djavax.net.ssl.trustStore=/data/glassfish/domains/prod/config/cacerts.jks -Dcom.sun.enterprise.security.httpsOutboundKeyAlias=s1as -Djava.security.auth.login.config=/data/glassfish/domains/prod/config/login.conf -DANTLR_USE_DIRECT_CLASS_LOADING=true -Dfelix.fileinstall.debug=1 -Dorg.glassfish.web.rfc2109_cookie_names_enforced=false -Djava.library.path=/usr/local/glassfishv3/glassfish/lib:/usr/java/jdk1.6.0_23/jre/lib/amd64/server:/usr/java/jdk1.6.0_23/jre/lib/amd64:/usr/java/jdk1.6.0_23/lib/amd64:/usr/local/oracle/db/10.2/EE/lib:/usr/local/oracle/db/10.2/EE/lib32:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib com.sun.enterprise.glassfish.bootstrap.ASMain -domainname prod -asadmin-args -host,,,localhost,,,port,,,4848,,,secure=true,,,terse=false,,,echo=false,,,interactive=true,,,start-domain,,,verbose=true,,,debug=false,,,-domaindir,,,/data/glassfish/domains,,,prod -instancename server -verbose true -debug false -asadmin-classpath /usr/local/glassfishv3/glassfish/modules/admin-cli.jar -asadmin-classname com.sun.enterprise.admin.cli.AsadminMain -upgrade false -type DAS -domaindir /data/glassfish/domains/prod -read-stdin true 06.01.2011 20:18:58 com.sun.enterprise.admin.launcher.GFLauncherLogger info INFO: Successfully launched in 12 msec. Launching GlassFish on Felix platform OSGiGlassFishRuntime: Initialized org.apache.felix.framework [0]

{it hangs here}

Environment

Linux 2.6.18-194.17.4.el5 #1 SMP Mon Oct 25 15:50:53 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux java version "1.6.0_23" Java(TM) SE Runtime Environment (build 1.6.0_23-b05) Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)

Affected Versions

[3.1]

[glassfishrobot]

• Issue Imported From: https://github.com/javaee/glassfish/issues/15462
• Original Issue Raised By:@glassfishrobot
• Original Issue Assigned To: @ssevozen
• Original Issue Closed By:@glassfishrobot

[glassfishrobot]

@glassfishrobot Commented @ssevozen said: Could you please let me know what was the starting version and did you remove the content of domain's osgi-cache directory before restarting?

[glassfishrobot]

@glassfishrobot Commented lft said: Version was 3.0.1 (build 22)

did you remove the content of domain's osgi-cache directory before restarting? noI didn't. I'll try it tomorrow.

[glassfishrobot]

@glassfishrobot Commented lft said: where is osgi-cache directory located?

[glassfishrobot]

@glassfishrobot Commented @ssevozen said: In your case it should be under /data/glassfish/domains/prod/osgi-cache.

[glassfishrobot]

@glassfishrobot Commented @ssevozen said: FWIW, I tried updating from OGS 3.0.1 to b36 and server restart worked for me even without osgi-cache cleanup although that's probably the side-effect of the fact that osgi-cache subdirectory is now called "Felix" and not "felix" as in 3.0.1 so these will not clash except on Windows... I also verified that installed content, except for default domain config, is identical to that of fresh b36 installation so everything seems to be fine from packaging/update center aspect.

Could you please tell me how exactly did you run the update? Which update center repository URL was used?

Also, please attach the output of 'pkg list' command run for this installation so that I can get full list of installed packages.

[glassfishrobot]

@glassfishrobot Commented lft said: I think yesterday I used repository stable.glassfish.org, because it's selected by default. Unfortunately I deleted .org.opensolaris,pkg directory yesterday during restoring of previous version. So today there is no updates available.

[glassfishrobot]

@glassfishrobot Commented lft said: I tried to copy this folder from another site with no success. pkg list shows nothing.

$ pkg list pkg: No image found.

[glassfishrobot]

@glassfishrobot Commented @ssevozen said: If you deleted .org.opensolaris,pkg directory, most of relevant information will unfortunately be gone. Do you still have any backup copy of that updated installation image?

I'll retry the update scenario with this information but to the best of my knowledge, you should not have seen any available updates to version 3.0.1 coming from stable repository since that's the last set of packages we published there...

[glassfishrobot]

@glassfishrobot Commented lft said:

Do you still have any backup copy of that updated installation image? Unfortunately, no.

you should not have seen any available updates to version 3.0.1 coming from stable repository since that's the last set of packages we published there... So there is no stable 3.1 version yet?

[glassfishrobot]

@glassfishrobot Commented lft said: Snjezana, can you tell me how can I restore .org.opensolaris,pkg directory, because "pkg image-update" & "pkg fix" didn't help me.

[glassfishrobot]

@glassfishrobot Commented lft said: By the way, on another server, which has not been updated and and "image" was just installed first time the "pkg list" command said the same:

$ pkg list pkg: No image found.

[glassfishrobot]

@glassfishrobot Commented lft said: I changed current directory to /usr/local/glassfishv3/bin and pkg seems to work now:

$ pwd /usr/local/glassfishv3/bin $ pkg list NAME (PUBLISHER) VERSION STATE UFIX felix 2.0.2-0 installed u--- glassfish-appclient 3.0.1-22 installed u--- glassfish-cmp 3.0.1-22 installed u--- glassfish-common 3.0.1-22 installed u--- glassfish-common-full 3.0.1-22 installed u--- glassfish-common-full-l10n 3.0.1-20.1 installed u--- glassfish-common-l10n 3.0.1-20.1 installed u--- glassfish-corba 3.0.0-41 installed u--- glassfish-corba-base 3.0.0-41 installed u--- glassfish-ejb 3.0.1-22 installed u--- glassfish-ejb-l10n 3.0.1-20.1 installed u--- glassfish-ejb-lite 3.0.1-22 installed u--- glassfish-ejb-lite-l10n 3.0.1-20.1 installed u--- glassfish-full-incorporation 3.0.1-22 installed u--- glassfish-full-profile 3.0.1-22 installed u--- glassfish-grizzly 1.9.18-10 installed u--- glassfish-grizzly-full 1.9.18-10 installed u--- glassfish-gui 3.0.1-22 installed u--- glassfish-gui-l10n 3.0.1-20.1 installed u--- glassfish-hk2 3.0.1-22 installed u--- glassfish-javahelp 2.0.2-0 installed u--- glassfish-jca 3.0.1-22 installed u--- glassfish-jca-l10n 3.0.1-20.1 installed u--- glassfish-jcdi 3.0.1-22 installed u--- glassfish-jdbc 3.0.1-22 installed u--- glassfish-jdbc-l10n 3.0.1-20.1 installed u--- glassfish-jms 3.0.1-22 installed u--- glassfish-jms-l10n 3.0.1-20.1 installed u--- glassfish-jpa 3.0.1-22 installed u--- glassfish-jsf 2.0.2-10 installed u--- glassfish-jta 3.0.1-22 installed u--- glassfish-jts 3.0.1-22 installed u--- glassfish-jts-l10n 3.0.1-20.1 installed u--- glassfish-management 3.0.1-22 installed u--- glassfish-nucleus 3.0.1-22 installed u--- glassfish-nucleus-l10n 3.0.1-20.1 installed u--- glassfish-registration 3.0.1-22 installed u--- glassfish-scripting 3.0.1-22 installed ---- glassfish-scripting-l10n 3.0.1-20.1 installed ---- glassfish-upgrade 3.0.1-22 installed u--- glassfish-upgrade-l10n 3.0.1-20.1 installed u--- glassfish-web 3.0.1-22 installed u--- glassfish-web-incorporation 3.0.1-22 installed u--- glassfish-web-l10n 3.0.1-20.1 installed u--- glassfish-web-profile 3.0.1-22 installed u--- javadb-client 10.5.3.0-1 installed u--- javadb-common 10.5.3.0-1 installed u--- javadb-core 10.5.3.0-1 installed u--- jersey 1.1.5-1.0 installed u--- metro 2.0.1-0 installed u--- mq-bin-exe 4.4.2-5.1 installed u--- mq-bin-sh 4.4.2-5.1 installed u--- mq-config-gf 4.4.2-5.1 installed u--- mq-core 4.4.2-5.1 installed u--- mq-locale 4.4.2-5.1 installed u--- mq-server 4.4.2-5.1 installed u--- pkg-java 1.122-38.2791 installed u---

[glassfishrobot]

@glassfishrobot Commented @ssevozen said: Yes, sorry, I forgot to mention that you do need to run pkg command using current working directory which is within the installation image directory in order to properly detect package information for that particular installation image....

Anyway, package listing is consistent with 3.0.1 installation and, as you noted, there are really no available 3.0.1 stable updates at this point. 3.1 promoted build packages are being published to dev.glassfish.org repository which is of lesser stability than stable.glassfish.org so those updates do not get automatically reported as available updates. In order to update from 3.0.1 to 3.1 promoted build, you would need to reconfigure updatetool to use dev.glassfish.org as preferred publisher and then apply all available updates.

So, unless I missed something in the thread, this is where we are at this point: it is not quite possible to reproduce what was the update path and procedure that caused your original domain restart problem. Package listing you provided is coming from your reinstalled 3.0.1 which is presumably behaving normally. If these statements are correct, I would be inclined to either downgrade the priority of this issue to "Minor" or to close it as unreproducible. If you run into similar situation again and provide full information on starting version and exact update procedure used, we should upgrade the priority or reopen the issue in order to address it in 3.1 release.

[glassfishrobot]

@glassfishrobot Commented @ssevozen said: As per my previous comment, I am closing the issue with "Cannot Reproduce" status. Please reopen with detailed description of update procedure if necessary.

[glassfishrobot]

@glassfishrobot Commented cistox said: I am sorry, but i have the same problem with latest GF 3.1 or even by upgrading from GF 3.0.1

[glassfishrobot]

@glassfishrobot Commented @ssevozen said: Could you please clarify this comment? I understand it to mean that you see the domain startup issue both with fresh GF 3.1 installation and after the upgrade from 3.0.1 to 3.1, is that correct?

If that is the case, your problem is not related to update procedure but to something else. Could you run 'asadmin start-domain' with -v option and see if you can get more information in verbose mode?

[glassfishrobot]

@glassfishrobot Commented cistox said: Hello, it is the GF Verifier I added using the update tool which is blocking the domain startup.

Using the update tool I removed the verifier and now I can start the domain again.

Please note i opened another issue about the Verifier as it seems the command-line parameters are not consistent with the effective requirements of the verifier, or anyway the verifier do not recognize very well its parameters (those mentioned on the command-line help).

Hope this helps.

Roberto

[glassfishrobot]

@glassfishrobot Commented This issue was imported from java.net JIRA GLASSFISH-15462

[glassfishrobot]

@glassfishrobot Commented Reported by lft

[glassfishrobot]

@glassfishrobot Commented Marked as cannot reproduce on Tuesday, January 11th 2011, 4:36:53 am