$expand tries to read much more data, than actually needed

[guai]

Here is the fix

[rzozmann]

Do you have a more concrete example/scenario to explain, when it happens that too much data is read?!

[guai]

It happens when data for ManyToOne dependencies being loaded. There is nothing limiting the request so all the data from the dependant table being loaded. When I'm requesting something like http://foo.bar/odata/ParentEntity?$top=3&$skip=0&$expand=* there is no need to request more than 3 rows for every ManyToOne attribute. I've added WHERE parent_table.id IN (...) clause.

[rzozmann]

So sorry for the delay... but the changes are breaking the test suite, so we have to inspect what happens.. With the current state the PR will have side effects, we have to manage; means still more work...

[guai]

I didn't manage to make tests pass even before my changes. I can look at what I broke if you give me some guides how to launch tests properly

[rzozmann]

I hope i will find also time for an deeper look. You can run the complete build + test suite via: clean verify -Duse-eclipselink -Ddisable.jetty=false -Dmaven.source.skip -Dmaven.javadoc.skip started in the root directory of local repository.

[guai]

In win 10 with oracle's java build 1.8.0_281-b09 and Apache Maven 3.6.3 the command you provided still does not pass against untouched branch origin/eXXcellent_adaptions from this repo. Can you describe me your env?

[INFO] Reactor Summary for OData-JPA-Adapter Root POM 0.52.1:
[INFO]
[INFO] OData-JPA-Adapter Root POM ......................... SUCCESS [  6.373 s]
[INFO] OData-JPA-Adapter implementation root .............. SUCCESS [  6.773 s]
[INFO] OData-JPA-Adapter data model configuration API ..... SUCCESS [ 13.651 s]
[INFO] OData-JPA-Adapter testmodel ........................ SUCCESS [  9.264 s]
[INFO] OData-JPA-Adapter metadata ......................... SUCCESS [ 22.433 s]
[INFO] OData-JPA-Adapter processor ........................ FAILURE [01:12 min]
[INFO] OData-JPA-Adapter addons ........................... SKIPPED
[INFO] OData-JPA-Adapter excelexport ...................... SKIPPED
[INFO] OData-JPA-Adapter Tooling Parent POM ............... SKIPPED
[INFO] OData-JPA-Adapter client api code generator for Java SKIPPED
[INFO] OData-JPA-Adapter examples ......................... SKIPPED
[INFO] OData-JPA-Adapter UI5 Client Example ............... SKIPPED
[INFO] OData-JPA-Adapter java-client-api-generator-example  SKIPPED
[INFO] OData-JPA-Adapter servlet-example .................. SKIPPED
[INFO] OData-JPA-Adapter coverage report module ........... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on project odata-jpa-processor: There are test failures.
[ERROR]
[ERROR] Please refer to D:\olingo-jpa-processor-v4-exxcellent\jpa\odata-jpa-processor\target\surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was cmd.exe /X /C "C:\jdk\jre\bin\java -javaagent:D:\\apache-maven-3.6.3\\repository\\org\\jacoco\\org.jacoco.agent\\0.8.5\\org.jacoco.agent-0.8.5-runtime.jar=destfile=D:\\olingo-jpa-processor-v4-exxcellent\\jpa\\odata-jpa-processor\\target\\jacoco.exec,append=true,inclnolocationclasses=true org.apache.maven.surefire.booter.ForkedBooter C:\Users\guai\AppData\Local\Temp\surefire3902702455596062375 2021-03-16T17-07-16_904-jvmRun1 surefire2822251614539718370tmp surefire_34118265438165058300tmp"
[ERROR] Process Exit Code: 0
[ERROR] Crashed tests:
[ERROR] org.apache.olingo.jpa.processor.core.query.TestJPACount
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was cmd.exe /X /C "C:\jdk\jre\bin\java -javaagent:D:\\apache-maven-3.6.3\\repository\\org\\jacoco\\org.jacoco.agent\\0.8.5\\org.jacoco.agent-0.8.5-runtime.jar=destfile=D:\\olingo-jpa-processor-v4-exxcellent\\jpa\\odata-jpa-processor\\target\\jacoco.exec,append=true,inclnolocationclasses=true org.apache.maven.surefire.booter.ForkedBooter C:\Users\guai\AppData\Local\Temp\surefire3902702455596062375 2021-03-16T17-07-16_904-jvmRun1 surefire2822251614539718370tmp surefire_34118265438165058300tmp"
[ERROR] Process Exit Code: 0
[ERROR] Crashed tests:
[ERROR] org.apache.olingo.jpa.processor.core.query.TestJPACount
[ERROR]         at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669)
[ERROR]         at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:282)
[ERROR]         at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:245)
[ERROR]         at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
[ERROR]         at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
[ERROR]         at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
[ERROR]         at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
[ERROR]         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
[ERROR]         at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
[ERROR]         at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
[ERROR]         at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
[ERROR]         at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
[ERROR]         at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
[ERROR]         at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
[ERROR]         at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
[ERROR]         at org.apache.maven.cli.MavenCli.execute(MavenCli.java:957)
[ERROR]         at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:289)
[ERROR]         at org.apache.maven.cli.MavenCli.main(MavenCli.java:193)
[ERROR]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR]         at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR]         at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
[ERROR]         at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
[ERROR]         at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
[ERROR]         at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
[ERROR]

[guai]

Looks like it was a maven bug can be fixed by adding <forkCount>0</forkCount> to maven-surefire-plugin's <configuration>. After that there are still some tests that fail:

[ERROR] Tests run: 8, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 35.928 s <<< FAILURE! - in org.apache.olingo.jpa.processor.transformation.excel.ExcelTransformationTest
[ERROR] testLoad(org.apache.olingo.jpa.processor.transformation.excel.ExcelTransformationTest)  Time elapsed: 32.941 s  <<< ERROR!
java.io.FileNotFoundException: target\test-load.xlsx (Системе не удается найти указанный путь)
        at org.apache.olingo.jpa.processor.transformation.excel.ExcelTransformationTest.testLoad(ExcelTransformationTest.java:200)

[ERROR] testSuccessfulFullExport(org.apache.olingo.jpa.processor.transformation.excel.ExcelTransformationTest)  Time elapsed: 0.307 s  <<< ERROR!
java.io.FileNotFoundException: target\test-full.xlsx (Системе не удается найти указанный путь)
        at org.apache.olingo.jpa.processor.transformation.excel.ExcelTransformationTest.testSuccessfulFullExport(ExcelTransformationTest.java:83)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR]   ExcelTransformationTest.testLoad:200 ? FileNotFound target\test-load.xlsx (Сис...
[ERROR]   ExcelTransformationTest.testSuccessfulFullExport:83 ? FileNotFound target\test...
[INFO]
[ERROR] Tests run: 15, Failures: 0, Errors: 2, Skipped: 0
[INFO]
[ERROR] There are test failures.

And I'm actually can't find neither test-load.xlsx nor test-full.xlsx in the repo...

[rzozmann]

And I'm actually can't find neither test-load.xlsx nor test-full.xlsx in the repo...

These excel-sheets are created by the test classes (testing the excel export), so they are not in the repository. I think your setup is correct (the build + tests are started). That you fail in 'processor' is expected; that is because i telled you that your changes have side effects.... I' normally running a linux environment, checking the Windows capabilities takes more time... What i can see: you have Maven 3 envionment: good; you have used Java 8: also good. The defaults of Maven should use 1 Thread, so a custom configuration (forkcount) shouldn't be neccessary... So your problems should be os dependent and no mistake by yourself... but that gives you no solace ;-)

[guai]

Are you saying that those errors about *.xlsx files caused by my changes? They are not. All the above listing I got on unchanged sources from origin/eXXcellent_adaptions. I wanted to have some baseline first and then look what my changes add to the picture.