Fix build and merge offline and other branches - Githubissues

orbeon / orbeon-forms

Orbeon Forms is an open source web forms solution. It includes an XForms engine, the Form Builder web-based form editor, and the Form Runner runtime.

http://www.orbeon.com/

GNU Lesser General Public License v2.1

514 stars 220 forks source link

Fix build and merge offline and other branches #4971

Closed ebruchez closed 2 years ago

ebruchez commented 3 years ago

[x] fix Proguard build
[x] fix tests
[x] check println(s"xxxx $jsonString")
[x] merge Excel import/export feature
[x] fix/reword commit messages especially those containing "xxx" and "TODO"
[x] cherry-pick rebases on pe branch
[x] figure out how to cherry-pick on master (CE) branch
- [x] create catalog of existing differences between CE/PE (files, etc.)
- [x] next step as needed (hopefully move towards a better solution)
[x] review 2020.1-pe branch for commits that are missing on master
[x] review wip-dependencies to see if some fixes need to be cherry-picked

ebruchez commented 3 years ago

ProGuard is an annoying bit. At first, it just failed. We had been running this with -dontwarn and -ignorewarnings, which obviously is a little bit dangerous but that's a slightly different story.

[x] I first tried excluding JARs that appeared to be related to the errors, but I got nowhere.
[x] upgraded ProGuard to 7.1.1
[x] opted for an additive process instead
- start with orbeon- JARs only
- progressively adding JAR files
- got to a point where I managed to get no errors, but the JAR was 10 MB (instead of 1-3 MB) (see #4148)
- got errors again, not sure why!
- used -whyareyoukeeping class org.orbeon.oxf.processor.EmailProcessor to get more visibility.
- export ANT_OPTS=-Xss2M helped with StackOverflowError in ant when doing this
[x] test orbeon-embedding.jar
[x] try excluding Saxon
- Doing so causes Proguard errors. We shouldn't need Saxon here, but is it worth spending a lot of time?
[ ] update JarJar rules
- [x] our code
- [ ] some sun.* and javax.* (why?)

ebruchez commented 3 years ago

Looked at https://github.com/yWorks/yGuard but it's unclear if it's better than ProGuard.

ebruchez commented 3 years ago

Managed to get to a 6.7 MB orbeon-embedding.jar without effort. Not sure why it was 10 MB before. But we clearly have some extra stuff that is most likely not used:

PageFlowControllerProcessor / PageFlowControllerProcessorBuilder
a lot of stuff in org.orbeon.oxf.processor
Saxon seems entirely present and accounts for around 1/3 to 1/4 of the classes
the Scala library appears to take almost 1/3 of the classes

On the plus side:

most of the xforms stuff is not there

ebruchez commented 3 years ago

Current sizes:

orbeon-embedding.jar: 7 MB
orbeon-proxy-portlet.jar: 7.1 MB
orbeon-xforms-filter.jar: 1.8 MB

ebruchez commented 3 years ago

Issues running tests:

[x] "Undo delete section, grid and control" use oracle.xml.jaxp.JXSAXParserFactory.setFeature
- Q: Why is this obsolete JAR in scope? See what we did on the Excel branch.
[x] "Referring to non-existent class scala.async.FutureStateMachine"
- downgraded scala-async
[x] fr:number: 1.2 is invalid if grouping separator is , fails (@avernet)
[x] replication test gets stuck, even locally
- [x] try running the requests from local browsers to see what happens
[x] "xxx GroupNode" logged
[x] build gets stuck on Travis even when disabling the replication test
- [x] try playing locally with heap settings

avernet commented 3 years ago

fr:number: 1.2 is invalid if grouping separator is , fails

I'll look at this one; this is most likely because it is now valid, after #4942. We have more tests for those cases in NumberSupportTest.scala, but I apparently missed the test in form-runner-components.xml.

avernet commented 3 years ago

fr:number: 1.2 is invalid if grouping separator is , fails

This is now fixed on master, on orbeon-forms and orbeon-forms-pe. @ebruchez, I'll let you cherry-pick the change on offline-support-1221-rebase, if needed.

ebruchez commented 3 years ago

Tests status:

local
- all tests pass with -J-Xmx4G in .sbtopts when OrbeonClientTest is disabled
- OrbeonClientTest either fails with "timeout expired"
  - timeout expires with 2.minutes
  - probably regression due to Headers.getAll change
  - timeout hangs after shutdown of imageA and trying to update window values
Travis CI
- build gets stuck when OrbeonClientTest is disabled

ebruchez commented 2 years ago

I went through diffs between CE and PE as of the master branch. It is not ideal how we are handling this, of course.

Properties a better solution could have:

differences are grouped together instead of spread around files
- for example they could be in one or more separate subprojects (form-runner-pe)
it's ok if we have a few files with diffs, for example
- property files
- build files
- one or two clear spots where, for example, specific components are registered
database support
- this needs to be modularized

ebruchez commented 2 years ago

Strategy for the fr.persistence changes:

list features that warrant these changes (encryption, etc.)
modularize on -rebase or PE master to have as much as possible separate files
- 1 commit is common code
- 1 commit is PE-only files
on CE master
- cp common code commit
- create new CE-only files

ebruchez commented 2 years ago

Features under fr.persistence:

encryption
Oracle / DB2 / SQL Server
- Provider has most diffs
- but some code in FlatView (feature is NOT PE-only as I thought)

ebruchez commented 2 years ago

[x] FlatView: move specific SQL to Provider and create PE provider functions
[x] FlatView: create CE provider functions in Provider
[x] modularize encryption code

ebruchez commented 2 years ago

modularize
- [x] PersistenceProxyProcessor.proxyReEncryptStatus
  - decided to keep as it doesn't hurt and is harder to factor out
- [x] CreateCols for id
- [x] FormRunnerHome.canReEncrypt
  - decided to keep
- [ ] Connect, DatasourceDescriptor, DDLTest, (for tests)
- [x] FormRunnerAPI
  - [x] embedForm() / destroyForm() (PE only)
  - [x] CE branch: update commit "For #4971: split out Form Runner embedding API (PE part)"
use new PE file
- [x] Permissions
- [x] PersistenceProxyProcessor
- [x] Reindex
- [x] CreateCols
- [x] CreateUpdateDelete
- [x] CRUD
- [x] LockUnlock
- [x] Read
- [x] commonPart
- [x] SearchLogic
- [x] RelationalCommon
- [x] RelationalUtils
- [x] FormRunnerPersistence
[x] FieldEncryption: adjust function params for CE
[x] universal-submission.xml
- changed

ebruchez commented 2 years ago

Files that remain different on PE and CE (not counting PE files that are entirely omitted on CE):

Controls form.xhtml
W9 data.xml
FormRunnerApp: to register xbl.HandwrittenSignature
page-flow.xml: reencrypt, schema
image-attachment.xbl
FieldEncryption
Provider
SimpleDataMigrationTest: disabled test on CE
OrbeonClientTest: disabled test on CE
facades: SignaturePad
properties-base.xml: oxf.resources.versioned
properties-form-builder.xml: handwritten signature, Email menu, workflow menu
properties-form-runner.xml: providers, autosave, PDF template/TIFF, lease
properties-internal.xml
processors.xml: reencrypt, XML Schema
home.xhtml: reencrypt
README.md
db tests (would be good to modularize)
- Connect
- DDLTest
- DatasourceDescriptor
build files
- build.sbt: CE vs. PE string
- build.xml: CE vs. PE string, orbeon-src (CHECK!), test-db, proxy-portlet, orbeon-embedding,
- .travis.yml: db stuff

ebruchez commented 2 years ago

It seems that he replication test now passes!

ebruchez commented 2 years ago

wip-dependencies handled. New branch wip-1192-dependencies created with WIP commits.