search

apache / parquet-java

Apache Parquet Java
https://parquet.apache.org/
Apache License 2.0
2.65k stars 1.41k forks source link

GH-3068: Remove `parquet-scala` #3063

Closed Fokko closed 1 day ago

Fokko commented 6 days ago

Rationale for this change

Based on the [DISCUSS] thread: https://lists.apache.org/thread/scdq9t2gvvs4glhq0qx4qcvfp62j793s

Closes #3068

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

pan3793 commented 6 days ago

Can we leave some words on README or somewhere to mention this removal?

Fokko commented 6 days ago

@pan3793 I noticed that the Scala DSL wasn't mentioned to the list, I've added it 👍

wgtmac commented 4 days ago

It seems that recent builds all fail with the following error:

[INFO] ---------------< org.apache.parquet:parquet-scala_2.12 >----------------
[INFO] Building Apache Parquet Scala 1.15.0-SNAPSHOT                    [16/18]
[INFO]   from parquet-scala/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] Downloading from scala-tools.org: https://scala-tools.org/repo-releases/net/alchim31/maven/scala-maven-plugin/4.9.1/scala-maven-plugin-4.9.1.pom
Warning:  Checksum validation failed, expected '<!DOCTYPE' (REMOTE_EXTERNAL) but is actually '3ee5a4fdb85dc0b9c676a21a40b6f1cc2d748a03' from scala-tools.org for https://scala-tools.org/repo-releases/net/alchim31/maven/scala-maven-plugin/4.9.1/scala-maven-plugin-4.9.1.pom
Warning:  Could not validate integrity of download from https://scala-tools.org/repo-releases/net/alchim31/maven/scala-maven-plugin/4.9.1/scala-maven-plugin-4.9.1.pom
org.eclipse.aether.transfer.ChecksumFailureException: Checksum validation failed, expected '<!DOCTYPE' (REMOTE_EXTERNAL) but is actually 'e0509cfe82a6ffad07497a39b82d6e0a82a5b1ac'
    at org.eclipse.aether.connector.basic.ChecksumValidator.validateExternalChecksums (ChecksumValidator.java:179)
    at org.eclipse.aether.connector.basic.ChecksumValidator.validate (ChecksumValidator.java:109)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask (BasicRepositoryConnector.java:484)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:414)
    at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get (BasicRepositoryConnector.java:260)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads (DefaultArtifactResolver.java:537)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:449)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:261)
    at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact (DefaultArtifactResolver.java:243)
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom (DefaultArtifactDescriptorReader.java:241)
    at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor (DefaultArtifactDescriptorReader.java:182)
    at org.eclipse.aether.internal.impl.DefaultRepositorySystem.readArtifactDescriptor (DefaultRepositorySystem.java:269)
    at org.apache.maven.plugin.internal.DefaultPluginDependenciesResolver.resolve (DefaultPluginDependenciesResolver.java:107)
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.lambda$getPluginDescriptor$0 (DefaultMavenPluginManager.java:183)
    at org.apache.maven.plugin.DefaultPluginDescriptorCache.lambda$get$0 (DefaultPluginDescriptorCache.java:72)
    at java.util.concurrent.ConcurrentHashMap.computeIfAbsent (ConcurrentHashMap.java:1660)
    at org.apache.maven.plugin.DefaultPluginDescriptorCache.get (DefaultPluginDescriptorCache.java:70)
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getPluginDescriptor (DefaultMavenPluginManager.java:181)
    at org.apache.maven.plugin.internal.DefaultMavenPluginManager.getMojoDescriptor (DefaultMavenPluginManager.java:271)
    at org.apache.maven.plugin.DefaultBuildPluginManager.getMojoDescriptor (DefaultBuildPluginManager.java:214)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleMappingDelegate.calculateLifecycleMappings (DefaultLifecycleMappingDelegate.java:101)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateLifecycleMappings (DefaultLifecycleExecutionPlanCalculator.java:239)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateMojoExecutions (DefaultLifecycleExecutionPlanCalculator.java:197)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateExecutionPlan (DefaultLifecycleExecutionPlanCalculator.java:119)
    at org.apache.maven.lifecycle.internal.DefaultLifecycleExecutionPlanCalculator.calculateExecutionPlan (DefaultLifecycleExecutionPlanCalculator.java:135)
    at org.apache.maven.lifecycle.internal.builder.BuilderCommon.resolveBuildPlan (BuilderCommon.java:93)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:100)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:903)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:280)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:203)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:255)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:201)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:361)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:314)
Warning:  Checksum validation failed, expected '<!DOCTYPE' (REMOTE_EXTERNAL) but is actually 'e0509cfe82a6ffad07497a39b82d6e0a82a5b1ac' from scala-tools.org for https://scala-tools.org/repo-releases/net/alchim31/maven/scala-maven-plugin/4.9.1/scala-maven-plugin-4.9.1.pom
[INFO] Downloaded from scala-tools.org: https://scala-tools.org/repo-releases/net/alchim31/maven/scala-maven-plugin/4.9.1/scala-maven-plugin-4.9.1.pom (23 kB at 30 kB/s)
Warning:  The POM for net.alchim31.maven:scala-maven-plugin:jar:4.9.1 is invalid, transitive dependencies (if any) will not be available: 1 problem was encountered while building the effective model
[FATAL] Non-parseable POM /home/runner/.m2/repository/net/alchim31/maven/scala-maven-plugin/4.9.1/scala-maven-plugin-4.9.1.pom: attribute value must start with quotation or apostrophe not M (position: START_DOCUMENT seen <!DOCTYPE html><html lang="en"  data-adblockkey=M... @1:50)  @ line 1, column 50

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Parquet Java 1.15.0-SNAPSHOT:
[INFO] 
[INFO] Apache Parquet Java ................................ SUCCESS [  9.926 s]
[INFO] Apache Parquet Format Structures ................... SUCCESS [  7.546 s]
[INFO] Apache Parquet Generator ........................... SUCCESS [  1.785 s]
[INFO] Apache Parquet Common .............................. SUCCESS [  3.653 s]
[INFO] Apache Parquet Encodings ........................... SUCCESS [  9.611 s]
[INFO] Apache Parquet Column .............................. SUCCESS [ 11.506 s]
[INFO] Apache Parquet Arrow ............................... SUCCESS [  2.374 s]
[INFO] Apache Parquet Jackson ............................. SUCCESS [  1.731 s]
[INFO] Apache Parquet Hadoop .............................. SUCCESS [ 15.833 s]
[INFO] Apache Parquet Avro ................................ SUCCESS [  9.508 s]
[INFO] Apache Parquet Benchmarks .......................... SUCCESS [ 16.404 s]
[INFO] Apache Parquet Command-line ........................ SUCCESS [  9.864 s]
[INFO] Apache Parquet Pig ................................. SUCCESS [  7.967 s]
[INFO] Apache Parquet Pig Bundle .......................... SUCCESS [  2.244 s]
[INFO] Apache Parquet Protobuf ............................ SUCCESS [ 10.107 s]
[INFO] Apache Parquet Scala ............................... FAILURE [  0.791 s]
[INFO] Apache Parquet Thrift .............................. SKIPPED
[INFO] Apache Parquet Hadoop Bundle ....................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:01 min
[INFO] Finished at: 2024-11-17T02:48:05Z
[INFO] ------------------------------------------------------------------------
Error:  Plugin net.alchim31.maven:scala-maven-plugin:4.9.1 or one of its dependencies could not be resolved:
Error:      1 problem was encountered while building the effective model
Error:  [FATAL] Non-parseable POM /home/runner/.m2/repository/net/alchim31/maven/scala-maven-plugin/4.9.1/scala-maven-plugin-4.9.1.pom: attribute value must start with quotation or apostrophe not M (position: START_DOCUMENT seen <!DOCTYPE html><html lang="en"  data-adblockkey=M... @1:50)  @ line 1, column 50
Error:  -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
Error: Process completed with exit code 1.

See: https://github.com/apache/parquet-java/actions/runs/11875285553/job/33092288845?pr=3064

Should we move forward with this PR?

Fokko commented 3 days ago

It looks like the Scala version is not available anymore. I would be in favor of removing the module.

I think there are no objections on the dev list either: https://lists.apache.org/thread/scdq9t2gvvs4glhq0qx4qcvfp62j793s

Fokko commented 3 days ago

@wgtmac Good point, I've added an issue 👍

Fokko commented 1 day ago

Let's move this forward to the CI works again, thanks for the review @pan3793 and @wgtmac 👍