Use of Grab statements within Groovy scripts causes error

russdanner commented 1 year ago

Duplicates

[X] I have searched the existing issues

Latest version

[X] The issue is in the develop branch
[x] The issue is in the latest released 4.1.x
[x] The issue is in the latest released 4.0.x
[x] The issue is in the latest released 3.1.x

Describe the issue

Use of a grab statement within scripts causes error:

Error was: failed to load settings from file:/home/russdanner/crafter-installs/next/craftercms/crafter-authoring/bin/grapeConfig.xml: 
Property 'http://javax.xml.xmlconstants/property/accessExternalSchema' is not recognized.

As a result, expected classes are not available and imports fail

Steps to reproduce

Steps:

Install commerce tools blueprint (uses grapes), API keys can be fake.
Visit homepage in preview
See the issue in logs and 500 error in the preview.

Relevant log output

No response

Screenshots and/or videos

No response

sumerjabri commented 1 year ago

Rolling back Ivy to 2.5.1 fixes the issue.

sumerjabri commented 1 year ago

@phuongnq please see why upgrading to Ivy 2.5.2 caused this issue. We need to upgrade to that version.

FYI, I also tried changing the grapeConfig.xml to:

<?xml version="1.0"?>
<ivysettings>
  <settings defaultResolver="downloadGrapes"/>
  <resolvers>
    <chain name="downloadGrapes">
      <filesystem name="cachedGrapes">
        <ivy pattern="${user.home}/.groovy/grapes/[organisation]/[module]/ivy-[revision].xml"/>
        <artifact pattern="${user.home}/.groovy/grapes/[organisation]/[module]/[type]s/[artifact]-[revision].[ext]"/>
      </filesystem>
      <!-- todo add 'endorsed groovy extensions' resolver here -->
      <ibiblio name="codehaus" root="http://repository.codehaus.org/" m2compatible="true"/>
      <ibiblio name="ibiblio" m2compatible="true"/>
      <ibiblio name="java.net2" root="http://download.java.net/maven/2/" m2compatible="true"/>
      <!-- this fixes the missing 2.1_3 version of cglib-nodep -->
      <ibiblio name="mirror" root="http://mirrors.ibiblio.org/maven2/" m2compatible="true"/>
    </chain>
  </resolvers>
</ivysettings>

and that didn't help.

Also tried:

Rolling back Groovy to 3.0.17
Changed the Ivy exclusions (see the history of pom.xml in Engine and Deployer)
Removing Xerces (which doesn't honor the settings accessExternalSchema (which we ask for))

phuongnq commented 1 year ago

@sumerjabri We have the easpi and tika packages also include xerces so I excluded them from engine pom.xml

sumerjabri commented 1 year ago

The same issue is likely present in deployer, will work on excluding Xerces there too.

sumerjabri commented 1 year ago

The problem ended up being: Xerces doesn't honor needed XML loading flags (like some key security flags). The solution was to exclude Xerces from Engine and Deployer since it's not necessary.

sumerjabri commented 1 year ago

@yacdaniel please validate this ticket and add it to our QA flow. Note it needs to be tested in 3.1, 4.0, and 4.1. You'll need to validate by having a project that pulls dependencies using Grape. Same in Deployer. Ping @russdanner for how to do this.

russdanner commented 1 year ago

Confirmed on my end.

Euquimides commented 1 year ago

Hi! Updated and rechecked this issue. Fix verified for Crafter v3.1.x, v4.0.8, v4.1.2 both CE and EE. Thank you! Closing this ticket.

russdanner commented 1 year ago

I just got the following error in 4.1.2-SNAPSHOT built today Local Ivy config file '/home/russdanner/crafter-installs/next/craftercms/crafter-authoring/bin/grapeConfig.xml' appears corrupt - ignoring it and using default config instead Error was: failed to load settings from file:/home/russdanner/crafter-installs/next/craftercms/crafter-authoring/bin/grapeConfig.xml: Property 'http://javax.xml.XMLConstants/property/accessExternalSchema' is not recognized.

With the following Grape import @Grab(group='io.github.http-builder-ng', module='http-builder-ng-core', version='1.0.4', initClass=false)

This Grab was from within a class loaded by a STUDIO plugin

russdanner commented 1 year ago

The following script works in Crafter Engine as a rest API but fails with the error above as a script in a plugin for Crafter Studio

@Grab(group='io.github.http-builder-ng', module='http-builder-ng-core', version='1.0.4', initClass=false)

import groovyx.net.http.HttpBuilder
import static groovyx.net.http.HttpBuilder.configure

def result = "foo" + HttpBuilder.configure { request.raw = "https://craftercms.com/" }.get()

return result

phuongnq commented 1 year ago

Other than the direct import of xerces, the daisydiff.jar also embedded the xerces internally which caused the issue for Studio. See the following screenshot:

daisydiff_xerces

There are some libraries that included the xerces while I check with jdeps:

# jdeps command result
Processing daisydiff.jar
   org.apache.html.dom.HTMLAnchorElementImpl          -> org.apache.xerces.dom.ElementImpl                  daisydiff.jar
   org.apache.html.dom.HTMLAppletElementImpl          -> org.apache.xerces.dom.ElementImpl                  daisydiff.jar
   org.apache.html.dom.HTMLAreaElementImpl            -> org.apache.xerces.dom.ElementImpl                  daisydiff.jar
   org.apache.html.dom.HTMLBRElementImpl              -> org.apache.xerces.dom.ElementImpl                  daisydiff.jar
   org.apache.html.dom.HTMLBaseElementImpl            -> org.apache.xerces.dom.ElementImpl                  daisydiff.jar
   org.apache.html.dom.HTMLBaseFontElementImpl        -> org.apache.xerces.dom.ElementImpl                  daisydiff.jar
   org.apache.html.dom.HTMLBodyElementImpl            -> org.apache.xerces.dom.ElementImpl                  daisydiff.jar
   ....
Processing xmlresolver-5.1.2.jar
   org.xmlresolver.XercesResolver                     -> org.apache.xerces.impl.XMLEntityDescription        not found
   org.xmlresolver.XercesResolver                     -> org.apache.xerces.impl.xs.XSDDescription           not found
   org.xmlresolver.XercesResolver                     -> org.apache.xerces.util.SAXInputSource              not found
   org.xmlresolver.XercesResolver                     -> org.apache.xerces.xni.XMLResourceIdentifier        not found
   org.xmlresolver.XercesResolver                     -> org.apache.xerces.xni.grammars.XMLDTDDescription   not found
   org.xmlresolver.XercesResolver                     -> org.apache.xerces.xni.parser.XMLEntityResolver     not found
   org.xmlresolver.XercesResolver                     -> org.apache.xerces.xni.parser.XMLInputSource        not found
Processing xom-1.3.8.jar
   nu.xom.Builder                                     -> org.apache.xerces.impl.Version                     not found
   nu.xom.XML1_0Parser                                -> org.apache.xerces.parsers.DTDConfiguration         not found
   nu.xom.XML1_0Parser                                -> org.apache.xerces.parsers.SAXParser                not found
   nu.xom.XML1_0Parser                                -> org.apache.xerces.xni.parser.XMLParserConfiguration not found

I have removed both in this PR: https://github.com/craftercms/studio/pull/3324

DaisyDiff: We can exclude the import by recompiling the jar file. This is also mentioned in the DaisyDiff README that:

To run Daisy Diff embedded in your application, you don't need the entire Jar file. A much smaller Jar file without Xerces and NekoHtml will suffice.

So I recompiled the jar to exclude xerces and NekoHtml.

xmlresolver: this is a dependency of net.sf.saxon in https://github.com/craftercms/studio/blob/develop/src/main/java/org/craftercms/studio/impl/v2/utils/XsltUtils.java#L50. But I don't see we include XercesResolver anywhere
xom-1.3.8: a dependency of org.owasp.esapi. If we don't see any problem in the engine, maybe it is fine to exclude xerces as we did for the Engine.

Other note:

Ivy also did a fix for this: https://github.com/apache/ant-ivy/commit/5d69c9a5447d32780e5c11f65a44b853647abdf4#diff-9232666cd01a28cd45ce5f9c43[…]7f435a924366e652093234395da05195. So they use trySetProperty instead of just setProperty which caused our error if the parser such as xerces does not support the property. But we will need their release of version 2.5.3 for this fix.

There are other temporary solutions besides excluding xerces to add the following parameter to JVM options:

-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl -Djavax.xml.transform.TransformerFactory=com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl

sumerjabri commented 1 year ago

@phuongnq let's get this in support/4.0.x as well.

yacdaniel commented 1 year ago

Verified fixed in 4.1.2-SNAPSHOT and 4.0.x

russdanner commented 2 months ago

This issue is reproducible in 4.2.0-Snapshot.

russdanner commented 2 months ago

Confirmed

craftercms / craftercms