Docker For Mac ONLY: Fedoragsearch refuses to start in isle-fedora container

[g7morris]

Issue not reproducible in other Host types

• Tested on a Vagrant / Virtualbox Ubuntu 16.04 VM Host on a MacOS
• Tested on a remote Digital Ocean server droplet running Ubuntu 16.04 LTS as the ISLE Host
• It did not manifest on any of the previous Alpha work (so far)

---

When testing the new alpha2 ISLE Docker images, on a local MacOS laptop running Docker For Mac, Fedoragsearch refuses to start in Tomcat on theisle-fedora` container.

All other containers work properly, even the installation of the Drupal site.

The only clue is within the Fedora container's tomcat logs, local_host_access.datetimestap.log

07-Feb-2018 20:39:17.563 SEVERE [http-nio-8080-exec-3] org.apache.catalina.core.StandardContext.filterStart Exception starting filter XmlUserfileFilter
java.lang.StackOverflowError
    at java.util.Hashtable.containsKey(Hashtable.java:336)
    at org.apache.log4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:36)
    at org.apache.log4j.LogManager.getLogger(LogManager.java:39)

org.apache.catalina.core.StandardContext.filterStart Exception starting filter EnforceAuthnFilter
java.lang.NoClassDefFoundError: Could not initialize class org.fcrepo.server.security.servletfilters.FilterSetup
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.catalina.core.DefaultInstanceManager.newInstance(DefaultInstanceManager.java:151)

---

My suspicion is how Docker For Mac handles the networking for localhost .

I did attempt to edit any value with localhost in the Fedoragsearch config files, swapping out localhost with fedora for all SOAP call related entries but to no avail as the app wouldn't start.

I wonder if there is an issue with access to Fedora by Fedoragsearch given the Could not initialize class org.fcrepo.server.security.servletfilters.FilterSetup entry however I don't have enough information at this time to proceed.

Is it possible that Fedora doesn't see the calls from Fedoragsearch to its repo as coming from the isle-fedora localhost and instead as something to be blocked by XACML or Spring (web.xml)/ filter ?

Posting this issue here for others to test.

Please note: MacOS users have a documented alternative method of using Vagrant and Virtualbox to host isle.localdomain locally on their laptops /workstations.

[br2490]

@g7morris I do not think this is just limited to Macs.

I just tried on an in-production docker host running Ubuntu 16.04. Then tried again on a VM using Ubuntu and it also failed. I'm not sure yet what the issue is nor why it works on RHEL and derivs just fine.

[DiegoPino]

I had some issues with that tomcat version (see docker updated to a new 8 2-3 weeks ago) and gsearch on one of our recently deployed 16.04 ubuntu islandora boxes. Removing one of the log translation .jar libraries (took me like 2 nights of debugging!) solved the issue. But i never got adore right (openURL could was not able to work with a mixed url+argument encoding scenario). Is adore working? And lastly, is islandora/fedora working fine? I had to tune the tomcat connector (valve) a few times because of random disconnections.

El El lun, 19 de feb. de 2018 a las 07:09, Benjamin Rosner < notifications@github.com> escribió:

@g7morris https://github.com/g7morris I do not think this is just limited to Macs.

I just tried on an in-production docker host running Ubuntu 16.04. Then tried again on a VM using Ubuntu and it also failed. I'm not sure yet what the issue is nor why it works on RHEL and derivs just fine.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Islandora-Collaboration-Group/ISLE/issues/97#issuecomment-366672922, or mute the thread https://github.com/notifications/unsubscribe-auth/AGn85yq1SrOMU_Q-suMUelQNjZZpfLaVks5tWWRdgaJpZM4R9poD .

-- Diego Pino Navarro Digital Repositories Developer Metropolitan New York Library Council (METRO)

[br2490]

@DiegoPino I'm going to check this hot tip out right now. thank you thank you.

Yes, everything is working including adore. Working branch/compose is currently a little more upstream @ https://github.com/BarnardArchives/barnard_isle_docker/tree/br2490-alpha2prodcompose. that is currently running at https://belle.benjaminrosner.com for testing.

Best, Ben

[g7morris]

@br2490 @DiegoPino I'm currently having a challenge with Adore and SSL though on one of the alphas. I'll have more to report tomorrow but would appreciate any further testing or insight in the meantime. Thanks very much for the updates.

[g7morris]

@DiegoPino Any tuning of tomcat tips etc., specific log files (jars) to exclude would also be appreciated!

[br2490]

@DiegoPino MANY THANKS!

mv log4j-over-slf4j-1.5.10.jar log4j-over-slf4j-1.5.10.jar.disabled results in OK - Started application at context path /fedoragsearch 👍 💯

Not 100% sure if that's the appropriate file to be move, but this is a huge step. (like, did I just disable logging? I will find out soon!)

edited to complete the thought.

[DiegoPino]

@br2490 cool, you found it! Yes, log4j-over-slf4j-1.5.10.jar generates an infinite loop there, trying to map back and forth the libraries. Wonder if we could actually ask current fedoragsearch maintainers to simply disregard that dependency at all.

[DiegoPino]

You did not disable logging at all. there are many logging libraries around and the one we are using is log4j, but what happens is that some libraries transform logs into a different format (like that one) but then there is another that maps back, so an infinite loop happens there. Why does that not fail in tomcat7? probably the order the libraries are started or, because tomcat provides its own conflicting with those. But you did not break anything. Have a production server (heavy load) running since dec 25th with that change applied. But I run (oldie here) everything under VM Ubuntus on AWS, not containerized (yet), so my experience varies.

Other thing: i had serious issues with Fedora disconnecting while ingesting objects with the built-in connector

In Server XML i changed it to

<Connector port="8080" protocol="org.apache.coyote.http11.Http11Nio2Protocol" connectionTimeout="-1" redirectPort="8443" enableLookups="true"  executor="tomcatThreadPool"  URIEncoding="UTF-8" keepAliveTimeout="-1"/> 

(change connectionTimeouts and keepalivetimeouts of course in production). After that fedora became a lot more stable. Not saying you shouldm but if you have randome ingest issues, maybe? Also maybe that is the reason why my adore never worked (i mean it works, but the way islandora passes the arguments when calling it is over-encoded or double encodede, not sure and fails responding with images, but starts fine). I moved on to Cantaloupe 3.x and very happy with it since then. But still, Ideploy adore for the Kakadu binaries.

[DiegoPino]

@g7morris about adore and SSL. My 2cents here. SSL is expensive on tomcat (and even when running super fast stuff like NGIX) so I really never had the need to connect Adore via SSL. My rule is SSL for endusers.. endpoints that require sensible data to pass, not this case maybe so my experience there varies. If your issues is in the way URLs are resolved (means you send something to Adore via Islandora and image fails and logs have a 500 error (internal!)= chances are you hit the same encoding issue i found. Means somehow islandora mixes argument v/s value v/s call in the URL encoding and when Tomcat (or adore, who knows who is doing this there) tries to split values and decode them, the call/arguments/URL ends being malformed for OpenURL. You could (not today, holiday!) if you have time put some debug statement and recompile adore and see how arguments are being matched, the OpenURL resolver is pretty simple code actually) but not sure if it is worth. Moving to IIIF seems more natural in this case at least.

[g7morris]

Hello @br2490 & @DiegoPino Thanks so much for the updates and the work on this.

I'd like to clarify the path where to remove the log4j-over-slf4j-1.5.10.jar file please.

Is it here? root@425656dc6519:/usr/local/tomcat/webapps/fedoragsearch/WEB-INF/lib# pwd /usr/local/tomcat/webapps/fedoragsearch/WEB-INF/lib

I ask as I'd like to rebuild the isle-fedora image with this file removed to stop further issues.

I still plan to do the merge today @br2490 but will rebuild post-merge etc.

[g7morris]

@DiegoPino Are there any clear instructions on how to use IIF / Cantaloupe with Islandora?

Additionally how to setup the Islandora Paged Content modules, Islandora openseadragon viewer /plugin and the IA Book reader?

Do you use SSL with them as well?

[DiegoPino]

@g7morris yes, it is that one. I would guess the best approach would be to stop tomcat, unzip the war manually, remove the lib and start Tomcat again. Just to keep logs clean of fatal failures.

[DiegoPino]

@g7morris about IIIF, there are no 100% clear instructions but could be done if you are interested (and can wait a few days) also probably better to open a different issue to follow up on those instructions.

[g7morris]

@DiegoPino That's good advice on the gsearch war / build. I'll follow those steps later today during the build.

[g7morris]

@DiegoPino Yes I think we'd be highly interested if you have time and energy to spare re the IIF / Cantaloupe resources. Opening ticket #111 to that effect and assigning to you. Thanks!

[DiegoPino]

@g7morris cool. Will work on this. /remind me about this on Friday if i don't come back to you before that (please)

[g7morris]

@DiegoPino & @br2490 I'm closing this ticket given that we've opened a new one #111 for Cantaloupe. This issue no longer persists in Docker For Mac or Ubuntu VMs and the fix / patch has been merged into master along with new images having been built.