search

yarl / pattypan

Upload files to Wikimedia Commons. The Spreadsheet Way.
https://commons.wikimedia.org/wiki/Commons:Pattypan
MIT License
56 stars 37 forks source link

Failure to complete upload #154

Open watty62 opened 2 years ago

watty62 commented 2 years ago

I've been testing 20.04 and now 22.02 on Mac OSX (Monterey 12.1).

In each case, Pattypan runs ok, allows me to generate a spreadsheet. I can validate the spreadsheet ok, and then try to upload. It authenticates and starts to upload but never finishes, even if I leave one file to upload for 2 hours.

This is the output. I hope that it is helpful. 

(base) ian@MacBook-Pro-2 Desktop % java --module-path "/Users/ian/javafx-sdk-17/lib" --add-modules javafx.web -jar pattypan.jar
Feb 07, 2022 7:01:15 PM pattypan.Main main

INFO: Wiki set as: commons.wikimedia.org

Protocol set as: https://
Script path set as: /w

Feb 07, 2022 7:01:15 PM pattypan.Main main
INFO: Operating System: Mac OS X
Pattypan Version: 22.02

Feb 07, 2022 7:01:59 PM Wiki getToken
INFO: Fetching URL https://commons.wikimedia.org/w/api.php?&maxlag=5&meta=tokens&format=xml&action=query&type=login

Feb 07, 2022 7:01:59 PM Wiki login
INFO: Fetching URL https://commons.wikimedia.org/w/api.php?&maxlag=5&format=xml&action=login

Feb 07, 2022 7:02:00 PM Wiki getUserInfo
INFO: Fetching URL https://commons.wikimedia.org/w/api.php?&usprop=editcount%7Cgroups%7Crights%7Cemailable%7Cblockinfo%7Cgender%7Cregistration&maxlag=5&format=xml&action=query&list=users

Feb 07, 2022 7:02:01 PM Wiki getSiteInfo
INFO: Fetching URL https://commons.wikimedia.org/w/api.php?&siprop=namespaces%7Cnamespacealiases%7Cgeneral%7Cextensions&maxlag=5&meta=siteinfo&format=xml&action=query

Feb 07, 2022 7:02:01 PM Wiki getSiteInfo
INFO: [commons.wikimedia.org] Successfully retrieved site info for commons.wikimedia.org

Feb 07, 2022 7:02:01 PM Wiki getUsers
INFO: [commons.wikimedia.org] Successfully retrieved user info for 1 users.

Feb 07, 2022 7:02:01 PM Wiki login
INFO: [commons.wikimedia.org] Successfully logged in as watty62, highLimit = false

Feb 07, 2022 7:02:06 PM Wiki getPageInfo
INFO: Fetching URL https://commons.wikimedia.org/w/api.php?&maxlag=5&prop=info&inprop=protection%7Cdisplaytitle%7Cwatchers&format=xml&action=query

Feb 07, 2022 7:02:06 PM Wiki getPageInfo
INFO: [commons.wikimedia.org] Successfully retrieved page info for 1 pages.

Feb 07, 2022 7:02:06 PM Wiki getPageInfo
INFO: Fetching URL https://commons.wikimedia.org/w/api.php?&maxlag=5&prop=info&inprop=protection%7Cdisplaytitle%7Cwatchers&format=xml&action=query

Feb 07, 2022 7:02:06 PM Wiki getPageInfo
INFO: [commons.wikimedia.org] Successfully retrieved page info for 1 pages.

Feb 07, 2022 7:02:06 PM Wiki getToken
INFO: Fetching URL https://commons.wikimedia.org/w/api.php?&maxlag=5&meta=tokens&format=xml&action=query&type=csrf

Feb 07, 2022 7:02:07 PM Wiki upload
INFO: Fetching URL https://commons.wikimedia.org/w/api.php?&maxlag=5&format=xml&action=upload

It goes no further than that.

fuzheado commented 2 years ago

Here's a data point that might be useful - I was able to successfully upload on my Mac (M1 mini) running Big Sur (MacOS version 11.6.2).

Java installed using "brew install java" using these directions: https://mkyong.com/java/how-to-install-java-on-mac-osx/

That is, after brew install, I made the java command usable by linking it like this: $ sudo ln -sfn /usr/local/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk

$ java -version openjdk version "15.0.2" 2021-01-19 OpenJDK Runtime Environment (build 15.0.2+7)

Running it like this: $ java -jar pattypan.jar

Running v22.02 pattypan, all worked out fine. Using two factor authentication, I used a bot password.

Here's the file: https://commons.wikimedia.org/wiki/File:Smithsonian-future-central-2021-12-26.jpg

Ambrosiani commented 2 years ago

update: using openjdk version 17.0.2 seems to work without errors!

Abbe98 commented 2 years ago

@Ambrosiani @fuzheado do you too need to load the module with --add-modules javafx.web?

Ambrosiani commented 2 years ago

@Abbe98 I need to add the module. When I run the plain java -jar pattypan.jar I get the following error. Seems like something in my java install isn't linked properly?

java -version
openjdk version "17.0.2" 2022-01-18
OpenJDK Runtime Environment Homebrew (build 17.0.2+0)
OpenJDK 64-Bit Server VM Homebrew (build 17.0.2+0, mixed mode, sharing)

feb. 08, 2022 9:53:07 FM com.sun.javafx.application.PlatformImpl startup
VARNING: Unsupported JavaFX configuration: classes were loaded from 'unnamed module @689eb690'
Loading library prism_es2 from resource failed: java.lang.UnsatisfiedLinkError: Can't load library: /Users/aronambrosiani/.openjfx/cache/17.0.2-ea/libprism_es2.dylib
java.lang.UnsatisfiedLinkError: Can't load library: /Users/aronambrosiani/.openjfx/cache/17.0.2-ea/libprism_es2.dylib
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2393)
    at java.base/java.lang.Runtime.load0(Runtime.java:755)
    at java.base/java.lang.System.load(System.java:1953)
    at com.sun.glass.utils.NativeLibLoader.installLibraryFromResource(NativeLibLoader.java:217)
    at com.sun.glass.utils.NativeLibLoader.loadLibraryFromResource(NativeLibLoader.java:197)
    at com.sun.glass.utils.NativeLibLoader.loadLibraryInternal(NativeLibLoader.java:138)
    at com.sun.glass.utils.NativeLibLoader.loadLibrary(NativeLibLoader.java:54)
    at com.sun.prism.es2.ES2Pipeline.lambda$static$0(ES2Pipeline.java:63)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
    at com.sun.prism.es2.ES2Pipeline.<clinit>(ES2Pipeline.java:52)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:375)
    at com.sun.prism.GraphicsPipeline.createPipeline(GraphicsPipeline.java:218)
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.init(QuantumRenderer.java:92)
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125)
    at java.base/java.lang.Thread.run(Thread.java:833)
Loading library prism_sw from resource failed: java.lang.UnsatisfiedLinkError: Can't load library: /Users/aronambrosiani/.openjfx/cache/17.0.2-ea/libprism_sw.dylib
java.lang.UnsatisfiedLinkError: Can't load library: /Users/aronambrosiani/.openjfx/cache/17.0.2-ea/libprism_sw.dylib
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2393)
    at java.base/java.lang.Runtime.load0(Runtime.java:755)
    at java.base/java.lang.System.load(System.java:1953)
    at com.sun.glass.utils.NativeLibLoader.installLibraryFromResource(NativeLibLoader.java:217)
    at com.sun.glass.utils.NativeLibLoader.loadLibraryFromResource(NativeLibLoader.java:197)
    at com.sun.glass.utils.NativeLibLoader.loadLibraryInternal(NativeLibLoader.java:138)
    at com.sun.glass.utils.NativeLibLoader.loadLibrary(NativeLibLoader.java:54)
    at com.sun.prism.sw.SWPipeline.lambda$static$0(SWPipeline.java:43)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
    at com.sun.prism.sw.SWPipeline.<clinit>(SWPipeline.java:42)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:375)
    at com.sun.prism.GraphicsPipeline.createPipeline(GraphicsPipeline.java:218)
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.init(QuantumRenderer.java:92)
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125)
    at java.base/java.lang.Thread.run(Thread.java:833)
Graphics Device initialization failed for :  es2, sw
Error initializing QuantumRenderer: no suitable pipeline found
java.lang.RuntimeException: java.lang.RuntimeException: Error initializing QuantumRenderer: no suitable pipeline found
    at com.sun.javafx.tk.quantum.QuantumRenderer.getInstance(QuantumRenderer.java:283)
    at com.sun.javafx.tk.quantum.QuantumToolkit.init(QuantumToolkit.java:254)
    at com.sun.javafx.tk.Toolkit.getToolkit(Toolkit.java:264)
    at com.sun.javafx.application.PlatformImpl.startup(PlatformImpl.java:291)
    at javafx.application.Platform.startup(Platform.java:113)
    at pattypan.Launcher.main(Unknown Source)
Caused by: java.lang.RuntimeException: Error initializing QuantumRenderer: no suitable pipeline found
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.init(QuantumRenderer.java:95)
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125)
    at java.base/java.lang.Thread.run(Thread.java:833)
Exception in thread "main" java.lang.RuntimeException: No toolkit found
    at com.sun.javafx.tk.Toolkit.getToolkit(Toolkit.java:276)
    at com.sun.javafx.application.PlatformImpl.startup(PlatformImpl.java:291)
    at javafx.application.Platform.startup(Platform.java:113)
    at pattypan.Launcher.main(Unknown Source)
Abbe98 commented 2 years ago

What error do you get if you run java -Djavafx.verbose=true -Dprism.verbose=true jar pattypan.jar and which architecture does your CPU use(In other words: which Mac do you use?)

Ambrosiani commented 2 years ago

M1 (arm) Mac Mini, Monterey 12.1. Same as fuzheado :)

verbose error:

java -Djavafx.verbose=true -Dprism.verbose=true -jar pattypan.jar
feb. 08, 2022 9:59:35 FM com.sun.javafx.application.PlatformImpl startup
VARNING: Unsupported JavaFX configuration: classes were loaded from 'unnamed module @689eb690'
Prism pipeline init order: es2 sw 
Using Double Precision Marlin Rasterizer
Using dirty region optimizations
Not using texture mask for primitives
Not forcing power of 2 sizes for textures
Using hardware CLAMP_TO_ZERO mode
Opting in for HiDPI pixel scaling
Prism pipeline name = com.sun.prism.es2.ES2Pipeline
Loading ES2 native library ... prism_es2
WARNING: java.lang.UnsatisfiedLinkError: Can't load library: /Applications/libprism_es2.dylib
Loading library prism_es2 from resource failed: java.lang.UnsatisfiedLinkError: Can't load library: /Users/aronambrosiani/.openjfx/cache/17.0.2-ea/libprism_es2.dylib
java.lang.UnsatisfiedLinkError: Can't load library: /Users/aronambrosiani/.openjfx/cache/17.0.2-ea/libprism_es2.dylib
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2393)
    at java.base/java.lang.Runtime.load0(Runtime.java:755)
    at java.base/java.lang.System.load(System.java:1953)
    at com.sun.glass.utils.NativeLibLoader.installLibraryFromResource(NativeLibLoader.java:217)
    at com.sun.glass.utils.NativeLibLoader.loadLibraryFromResource(NativeLibLoader.java:197)
    at com.sun.glass.utils.NativeLibLoader.loadLibraryInternal(NativeLibLoader.java:138)
    at com.sun.glass.utils.NativeLibLoader.loadLibrary(NativeLibLoader.java:54)
    at com.sun.prism.es2.ES2Pipeline.lambda$static$0(ES2Pipeline.java:63)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
    at com.sun.prism.es2.ES2Pipeline.<clinit>(ES2Pipeline.java:52)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:375)
    at com.sun.prism.GraphicsPipeline.createPipeline(GraphicsPipeline.java:218)
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.init(QuantumRenderer.java:92)
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125)
    at java.base/java.lang.Thread.run(Thread.java:833)
GraphicsPipeline.createPipeline failed for com.sun.prism.es2.ES2Pipeline
java.lang.UnsatisfiedLinkError: no prism_es2 in java.library.path: /Users/aronambrosiani/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2429)
    at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:818)
    at java.base/java.lang.System.loadLibrary(System.java:1989)
    at com.sun.glass.utils.NativeLibLoader.loadLibraryInternal(NativeLibLoader.java:166)
    at com.sun.glass.utils.NativeLibLoader.loadLibrary(NativeLibLoader.java:54)
    at com.sun.prism.es2.ES2Pipeline.lambda$static$0(ES2Pipeline.java:63)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
    at com.sun.prism.es2.ES2Pipeline.<clinit>(ES2Pipeline.java:52)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:375)
    at com.sun.prism.GraphicsPipeline.createPipeline(GraphicsPipeline.java:218)
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.init(QuantumRenderer.java:92)
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125)
    at java.base/java.lang.Thread.run(Thread.java:833)
*** Fallback to Prism SW pipeline
Prism pipeline name = com.sun.prism.sw.SWPipeline
WARNING: java.lang.UnsatisfiedLinkError: Can't load library: /Applications/libprism_sw.dylib
Loading library prism_sw from resource failed: java.lang.UnsatisfiedLinkError: Can't load library: /Users/aronambrosiani/.openjfx/cache/17.0.2-ea/libprism_sw.dylib
java.lang.UnsatisfiedLinkError: Can't load library: /Users/aronambrosiani/.openjfx/cache/17.0.2-ea/libprism_sw.dylib
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2393)
    at java.base/java.lang.Runtime.load0(Runtime.java:755)
    at java.base/java.lang.System.load(System.java:1953)
    at com.sun.glass.utils.NativeLibLoader.installLibraryFromResource(NativeLibLoader.java:217)
    at com.sun.glass.utils.NativeLibLoader.loadLibraryFromResource(NativeLibLoader.java:197)
    at com.sun.glass.utils.NativeLibLoader.loadLibraryInternal(NativeLibLoader.java:138)
    at com.sun.glass.utils.NativeLibLoader.loadLibrary(NativeLibLoader.java:54)
    at com.sun.prism.sw.SWPipeline.lambda$static$0(SWPipeline.java:43)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
    at com.sun.prism.sw.SWPipeline.<clinit>(SWPipeline.java:42)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:375)
    at com.sun.prism.GraphicsPipeline.createPipeline(GraphicsPipeline.java:218)
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.init(QuantumRenderer.java:92)
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125)
    at java.base/java.lang.Thread.run(Thread.java:833)
GraphicsPipeline.createPipeline failed for com.sun.prism.sw.SWPipeline
java.lang.UnsatisfiedLinkError: no prism_sw in java.library.path: /Users/aronambrosiani/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2429)
    at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:818)
    at java.base/java.lang.System.loadLibrary(System.java:1989)
    at com.sun.glass.utils.NativeLibLoader.loadLibraryInternal(NativeLibLoader.java:166)
    at com.sun.glass.utils.NativeLibLoader.loadLibrary(NativeLibLoader.java:54)
    at com.sun.prism.sw.SWPipeline.lambda$static$0(SWPipeline.java:43)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
    at com.sun.prism.sw.SWPipeline.<clinit>(SWPipeline.java:42)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:375)
    at com.sun.prism.GraphicsPipeline.createPipeline(GraphicsPipeline.java:218)
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.init(QuantumRenderer.java:92)
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125)
    at java.base/java.lang.Thread.run(Thread.java:833)
Graphics Device initialization failed for :  es2, sw
Error initializing QuantumRenderer: no suitable pipeline found
java.lang.RuntimeException: java.lang.RuntimeException: Error initializing QuantumRenderer: no suitable pipeline found
    at com.sun.javafx.tk.quantum.QuantumRenderer.getInstance(QuantumRenderer.java:283)
    at com.sun.javafx.tk.quantum.QuantumToolkit.init(QuantumToolkit.java:254)
    at com.sun.javafx.tk.Toolkit.getToolkit(Toolkit.java:264)
    at com.sun.javafx.application.PlatformImpl.startup(PlatformImpl.java:291)
    at javafx.application.Platform.startup(Platform.java:113)
    at pattypan.Launcher.main(Unknown Source)
Caused by: java.lang.RuntimeException: Error initializing QuantumRenderer: no suitable pipeline found
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.init(QuantumRenderer.java:95)
    at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125)
    at java.base/java.lang.Thread.run(Thread.java:833)
Exception in thread "main" java.lang.RuntimeException: No toolkit found
    at com.sun.javafx.tk.Toolkit.getToolkit(Toolkit.java:276)
    at com.sun.javafx.application.PlatformImpl.startup(PlatformImpl.java:291)
    at javafx.application.Platform.startup(Platform.java:113)
    at pattypan.Launcher.main(Unknown Source)
Abbe98 commented 2 years ago

I built and tested Pattypan for x64 thinking that Mac remained backward compatible across the many changes of architecture, but maybe the M1 broke that.

That said the original issue appears to be something different, but this is worth keeping in mind as we might need separate builds for MacOS.

watty62 commented 2 years ago

What error do you get if you run java -Djavafx.verbose=true -Dprism.verbose=true jar pattypan.jar and which architecture does your CPU use(In other words: which Mac do you use?)

(base) ian@MacBook-Pro-2 Desktop % java -Djavafx.verbose=true -Dprism.verbose=true jar pattypan.jar Error: Could not find or load main class jar Caused by: java.lang.ClassNotFoundException: jar

My mac is pre-M1: Model Name: MacBook Pro Model Identifier: MacBookPro15,1 Processor Name: 6-Core Intel Core i9 Processor Speed: 2.9 GHz Number of Processors: 1 Total Number of Cores: 6 L2 Cache (per Core): 256 KB L3 Cache: 12 MB Hyper-Threading Technology: Enabled Memory: 32 GB

Java version: openjdk 17.0.2 2022-01-18 OpenJDK Runtime Environment Homebrew (build 17.0.2+0) OpenJDK 64-Bit Server VM Homebrew (build 17.0.2+0, mixed mode, sharing)

And I have javafx-sdk-17.0.2 for X64.

Let me know if I can help more.

Ambrosiani commented 2 years ago

@watty62 there's a missing hyphen in that command, try java -Djavafx.verbose=true -Dprism.verbose=true -jar pattypan.jar

watty62 commented 2 years ago

Thanks @Ambrosiani

It runs Pattypan and gives the following output

(base) ian@MacBook-Pro-2 Desktop % java -Djavafx.verbose=true -Dprism.verbose=true -jar pattypan.jar
Feb 08, 2022 9:46:18 AM com.sun.javafx.application.PlatformImpl startup
WARNING: Unsupported JavaFX configuration: classes were loaded from 'unnamed module @4d937939'
Prism pipeline init order: es2 sw 
Using Double Precision Marlin Rasterizer
Using dirty region optimizations
Not using texture mask for primitives
Not forcing power of 2 sizes for textures
Using hardware CLAMP_TO_ZERO mode
Opting in for HiDPI pixel scaling
Prism pipeline name = com.sun.prism.es2.ES2Pipeline
Loading ES2 native library ... prism_es2
WARNING: java.lang.UnsatisfiedLinkError: Can't load library: /Users/ian/Desktop/libprism_es2.dylib
Loaded library /libprism_es2.dylib from resource
    succeeded.
GLFactory using com.sun.prism.es2.MacGLFactory
(X) Got class = class com.sun.prism.es2.ES2Pipeline
Initialized prism pipeline: com.sun.prism.es2.ES2Pipeline
JavaFX: using com.sun.javafx.tk.quantum.QuantumToolkit
WARNING: java.lang.UnsatisfiedLinkError: Can't load library: /Users/ian/Desktop/libglass.dylib
Loaded library /libglass.dylib from resource
Maximum supported texture size: 16384
Maximum texture size clamped to 4096
Non power of two texture support = true
Maximum number of vertex attributes = 16
Maximum number of uniform vertex components = 4096
Maximum number of uniform fragment components = 4096
Maximum number of varying components = 128
Maximum number of texture units usable in a vertex shader = 16
Maximum number of texture units usable in a fragment shader = 16
Graphics Vendor: ATI Technologies Inc.
       Renderer: AMD Radeon Pro 560X OpenGL Engine
        Version: 2.1 ATI-4.7.101
 vsync: true vpipe: true
WARNING: java.lang.UnsatisfiedLinkError: Can't load library: /Users/ian/Desktop/libjavafx_font.dylib
Loaded library /libjavafx_font.dylib from resource
Feb 08, 2022 9:46:20 AM pattypan.Main main
INFO: Wiki set as: commons.wikimedia.org
Protocol set as: https://
Script path set as: /w
Feb 08, 2022 9:46:20 AM pattypan.Main main
INFO: Operating System: Mac OS X
Pattypan Version: 22.02
ES2ResourceFactory: Prism - createStockShader: FillPgram_Color.frag
ES2ResourceFactory: Prism - createStockShader: Solid_TextureFirstPassLCD.frag
ES2ResourceFactory: Prism - createStockShader: Solid_TextureSecondPassLCD.frag
new alphas with length = 49152
ES2ResourceFactory: Prism - createStockShader: Texture_Color.frag
ES2ResourceFactory: Prism - createStockShader: DrawRoundRect_Color.frag
ES2ResourceFactory: Prism - createStockShader: Solid_TextureRGB.frag
watty62 commented 2 years ago

Today I installed Virtual Box and configured an Ubuntu 20 VM.

I installed Open Java JDK and Open JavaFX 17.0.2

I ran Pattypan and tried to upload two files.

I hit exactly the same issue as reported in the original issue above. ie Pattypan runs ok, allows me to generate a spreadsheet. I can validate the spreadsheet ok, and then try to upload. It authenticates and starts to upload the first file but never finishes.

Whereas the OS has changed from Mac OSX 12 to Ubuntu 20.04.3, the Java and JavaFX versions are the same.

Running the command from @Ambrosiani above java -Djavafx.verbose=true -Dprism.verbose=true -jar pattypan.jar i get the output below.

I hope that this helps!

Ian

watty@watty-VirtualBox:~/Documents$ java -Djavafx.verbose=true -Dprism.verbose=true -jar pattypan.jar
Feb 08, 2022 7:25:12 PM com.sun.javafx.application.PlatformImpl startup
WARNING: Unsupported JavaFX configuration: classes were loaded from 'unnamed module @685f4c2e'
Prism pipeline init order: es2 sw 
Using Double Precision Marlin Rasterizer
Using dirty region optimizations
Not using texture mask for primitives
Not forcing power of 2 sizes for textures
Using hardware CLAMP_TO_ZERO mode
Opting in for HiDPI pixel scaling
Prism pipeline name = com.sun.prism.es2.ES2Pipeline
Loading ES2 native library ... prism_es2
WARNING: java.lang.UnsatisfiedLinkError: Can't load library: /home/watty/Documents/libprism_es2.so
Loaded library /libprism_es2.so from resource
    succeeded.
GLFactory using com.sun.prism.es2.X11GLFactory
(X) Got class = class com.sun.prism.es2.ES2Pipeline
Failed Graphics Hardware Qualifier check.
System GPU doesn't meet the es2 pipe requirement
GraphicsPipeline.createPipeline: error initializing pipeline com.sun.prism.es2.ES2Pipeline
*** Fallback to Prism SW pipeline
Prism pipeline name = com.sun.prism.sw.SWPipeline
WARNING: java.lang.UnsatisfiedLinkError: Can't load library: /home/watty/Documents/libprism_sw.so
Loaded library /libprism_sw.so from resource
(X) Got class = class com.sun.prism.sw.SWPipeline
Initialized prism pipeline: com.sun.prism.sw.SWPipeline
JavaFX: using com.sun.javafx.tk.quantum.QuantumToolkit
WARNING: java.lang.UnsatisfiedLinkError: Can't load library: /home/watty/Documents/libglass.so
Loaded library /libglass.so from resource
WARNING: java.lang.UnsatisfiedLinkError: Can't load library: /home/watty/Documents/libglassgtk3.so
Loaded library /libglassgtk3.so from resource
 vsync: true vpipe: false
WARNING: java.lang.UnsatisfiedLinkError: Can't load library: /home/watty/Documents/libjavafx_font.so
Loaded library /libjavafx_font.so from resource
Feb 08, 2022 7:25:13 PM pattypan.Main main
INFO: Wiki set as: commons.wikimedia.org
Protocol set as: https://
Script path set as: /w
Feb 08, 2022 7:25:13 PM pattypan.Main main
INFO: Operating System: Linux
Pattypan Version: 22.02
WARNING: java.lang.UnsatisfiedLinkError: Can't load library: /home/watty/Documents/libjavafx_font_freetype.so
Loaded library /libjavafx_font_freetype.so from resource
WARNING: java.lang.UnsatisfiedLinkError: Can't load library: /home/watty/Documents/libjavafx_font_pango.so
Loaded library /libjavafx_font_pango.so from resource
Abbe98 commented 2 years ago

@watty62 it indeed looks like the upload of the first image never finishes. Would you be able to share the first file? Some files just don't play nice with Pattypan nor Commons, especially TIFF files but it happens to other formats as well.

watty62 commented 2 years ago

@Abbe98 I didn't think of that as a common issue - ie using the same file each time.

Here it is: https://www.dropbox.com/s/1f6eln8cfxf9cki/20220126_125956.jpg?dl=0

Thanks

Ian

fuzheado commented 2 years ago

@Ambrosiani @fuzheado do you too need to load the module with --add-modules javafx.web?

Yes, to be clear, when it worked "out of the box" with OpenJDK 15.0.2 and found JavaFX automatically I think it was using an old installation from Oracle, so it was bundled in.

When I used "brew install java" with is openjdk 17.0.2, I need to manually download JavaFX and include the module path and module.

watty62 commented 2 years ago

@Abbe98 using a different photo I was able to upload via Pattypan on the Mac.

https://commons.wikimedia.org/wiki/File:Sir_George_Skene_of_Wester_Fintray_and_Rubislaw,_Provost_ofAberdeen(1676-85)_-_ABDCC001018.jpg

Two of my own images failed but this PD / CC0 one worked. It is of much smaller dimensions, whereas my own were 4,032 x 1,816 px and 3MB+

I am going to try another batch later today or tomorrow. I will note the outcomes here.

annalauraw commented 2 years ago

I have the same issue: Pattypan 22.02 starts fine and gets stuck at the upload of the first image. This happened in Windows 10, Ubuntu 21.10 and Ubuntu 18.04. I had the same issue with the experimental releases 3 and 4, as documented here: https://github.com/yarl/pattypan/issues/145, though @MER-C was able to upload my images to test.wikipedia.org without problems.

I tried with OpenJDK 11 and 17.

Additional info: On Windows 10, when I start 22.02 with java -jar pattypan.jar, I get the following warning: Unsupported JavaFX configuration: classes were loaded from 'unnamed module @6193b845' When I start it with java --module-path %PATH_TO_FX% --add-modules javafx.web -jar pattypan_22.jar, I get no such warning, but the behaviour is the same.

annalauraw commented 2 years ago

Here's my output for $JAVA_HOME/bin/java -Djavafx.verbose=true -Dprism.verbose=true --module-path $PATH_TO_FX --add-modules javafx.web -jar Downloads/pattypan_22.jar:

Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-console-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-console-l1-2-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-datetime-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-debug-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-errorhandling-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-file-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-file-l1-2-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-file-l2-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-handle-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-heap-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-interlocked-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-libraryloader-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-localization-l1-2-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-memory-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-namedpipe-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-processenvironment-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-processthreads-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-processthreads-l1-1-1.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-profile-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-rtlsupport-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-string-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-synch-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-synch-l1-2-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-sysinfo-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-timezone-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-core-util-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-conio-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-convert-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-environment-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-filesystem-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-heap-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-locale-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-math-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-multibyte-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-private-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-process-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-runtime-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-stdio-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-string-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-time-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\api-ms-win-crt-utility-l1-1-0.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\ucrtbase.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\vcruntime140.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\vcruntime140_1.dll from relative path Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\msvcp140.dll from relative path Prism pipeline init order: d3d sw Using Double Precision Marlin Rasterizer Using dirty region optimizations Not using texture mask for primitives Not forcing power of 2 sizes for textures Using hardware CLAMP_TO_ZERO mode Opting in for HiDPI pixel scaling Prism pipeline name = com.sun.prism.d3d.D3DPipeline Loading D3D native library ... Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\prism_d3d.dll from relative path succeeded. D3DPipelineManager: Created D3D9Ex device Direct3D initialization succeeded (X) Got class = class com.sun.prism.d3d.D3DPipeline Initialized prism pipeline: com.sun.prism.d3d.D3DPipeline JavaFX: using com.sun.javafx.tk.quantum.QuantumToolkit Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\glass.dll from relative path Maximum supported texture size: 8192 Maximum texture size clamped to 4096 OS Information: Windows version 10.0 build 19044 D3D Driver Information: Intel(R) UHD Graphics \.\DISPLAY1 Driver igdumdim64.dll, version 30.0.100.9955 Pixel Shader version 3.0 Device : ven_8086, dev_9B41, subsys_22AD17AA Max Multisamples supported: 4 vsync: true vpipe: true Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\javafx_font.dll from relative path Feb. 10, 2022 3:12:55 PM pattypan.Main main INFORMATION: Wiki set as: commons.wikimedia.org Protocol set as: https:// Script path set as: /w Feb. 10, 2022 3:12:55 PM pattypan.Main main INFORMATION: Operating System: Windows 10 Pattypan Version: 22.02 new alphas with length = 16384 new alphas with length = 20480 PPSRenderer: scenario.effect - createShader: LinearConvolveShadow_4 Loaded C:\Java\javafx-sdk-17.0.1\lib..\bin\javafx_iio.dll from relative path Growing pool D3D Vram Pool target to 79▒368▒512 Growing pool D3D Vram Pool target to 97▒093▒604 Growing pool D3D Vram Pool target to 111▒773▒668 Feb. 10, 2022 3:14:14 PM Wiki getToken INFORMATION: Fetching URL https://commons.wikimedia.org/w/api.php?&maxlag=5&meta=tokens&format=xml&action=query&type=login Feb. 10, 2022 3:14:14 PM org.wikipedia.Wiki checkLag WARNUNG: Current database lag 12 s exceeds maxlag of 5 s, waiting 5 s. Feb. 10, 2022 3:14:19 PM Wiki getToken INFORMATION: Fetching URL https://commons.wikimedia.org/w/api.php?&maxlag=5&meta=tokens&format=xml&action=query&type=login Feb. 10, 2022 3:14:19 PM org.wikipedia.Wiki checkLag WARNUNG: Current database lag 17 s exceeds maxlag of 5 s, waiting 5 s. Feb. 10, 2022 3:14:24 PM Wiki getToken INFORMATION: Fetching URL https://commons.wikimedia.org/w/api.php?&maxlag=5&meta=tokens&format=xml&action=query&type=login Feb. 10, 2022 3:14:25 PM org.wikipedia.Wiki checkLag WARNUNG: Current database lag 22 s exceeds maxlag of 5 s, waiting 5 s. Feb. 10, 2022 3:14:30 PM Wiki getToken INFORMATION: Fetching URL https://commons.wikimedia.org/w/api.php?&maxlag=5&meta=tokens&format=xml&action=query&type=login Feb. 10, 2022 3:14:30 PM org.wikipedia.Wiki checkLag WARNUNG: Current database lag 6 s exceeds maxlag of 5 s, waiting 5 s. Feb. 10, 2022 3:14:35 PM Wiki getToken INFORMATION: Fetching URL https://commons.wikimedia.org/w/api.php?&maxlag=5&meta=tokens&format=xml&action=query&type=login Feb. 10, 2022 3:14:35 PM Wiki login INFORMATION: Fetching URL https://commons.wikimedia.org/w/api.php?&maxlag=5&format=xml&action=login Feb. 10, 2022 3:14:36 PM Wiki getUserInfo INFORMATION: Fetching URL https://commons.wikimedia.org/w/api.php?&usprop=editcount%7Cgroups%7Crights%7Cemailable%7Cblockinfo%7Cgender%7Cregistration&maxlag=5&format=xml&action=query&list=users Feb. 10, 2022 3:14:36 PM Wiki getSiteInfo INFORMATION: Fetching URL https://commons.wikimedia.org/w/api.php?&siprop=namespaces%7Cnamespacealiases%7Cgeneral%7Cextensions&maxlag=5&meta=siteinfo&format=xml&action=query Feb. 10, 2022 3:14:37 PM Wiki getSiteInfo INFORMATION: [commons.wikimedia.org] Successfully retrieved site info for commons.wikimedia.org Feb. 10, 2022 3:14:37 PM Wiki getUsers INFORMATION: [commons.wikimedia.org] Successfully retrieved user info for 1 users. Feb. 10, 2022 3:14:37 PM Wiki login INFORMATION: [commons.wikimedia.org] Successfully logged in as Zentralbibliothek Z▒rich, highLimit = false Feb. 10, 2022 3:14:47 PM Wiki getPageInfo INFORMATION: Fetching URL https://commons.wikimedia.org/w/api.php?&maxlag=5&prop=info&inprop=protection%7Cdisplaytitle%7Cwatchers&format=xml&action=query

Abbe98 commented 2 years ago

@annalauraw it appears that this issue is caused by the files in question. Could you too share the ones which fail?

One solution that could be tried is to convert the problematic images to another format and back agian.

annalauraw commented 2 years ago

Here you can find the failing files: https://drive.switch.ch/index.php/s/nXiYUN34qGfhQsZ. It's the same files that @MER-C was able to upload to test.wikipedia.org without problems, as mentionied in https://github.com/yarl/pattypan/issues/145#issuecomment-1029271082.

In our workflow, we download the images from our digital library platform in .jpg format using a Python script (download image as stream via IIIF API and save to .jpg file). Could the problem lie in this process?

annalauraw commented 2 years ago

I tried to convert the .jpg files to png format and back, but the upload still doesn't work. Could spaces or other special characters in the file name (not in image path on local disk) be a problem? For instance CH-ZBZ-[Ausbeüte der Böcken, bey Belagerung der Statt Zürich]-991012222739705501.jpg

annalauraw commented 2 years ago

Turns out the brackets in the file name were causing the problem! Without the brackets, it worked. May be useful to know for other GLAM users since librarians tend to put titles which are found outside of the image resource in square brackets.

Abbe98 commented 2 years ago

@annalauraw great, the March release will contain #133 that should introduce a warning for such issues.

watty62 commented 2 years ago

I can confirm that a file name with an accidental carriage return in it also fails to upload (whereas those containing double quotes or forward slashes are fixed in the upload process)

MER-C commented 2 years ago

https://www.mediawiki.org/wiki/Help:Bad_title#regex

bettinacsmith commented 2 years ago

I am having what seems to be the same issue with 22.03 - I'm able to generate a spreadsheet but when I go to upload it will upload a few files (as few as one and up to maybe 15) and then stops. I've been stopping the upload when it seems stalled out, reordering the files in the upload spreadsheet and trying again. It will do the same thing - upload a few files and then stop. It doesn't seem to be tied to one particular file because after reordering, the file that it stopped on the previous time will be uploaded without issue. I'm operating in Windows 10.

Abbe98 commented 2 years ago

@bettinacsmith could you provide additional information about the files you are uploading?

Wikimedia Commons uploads commonly and irregularly fail for a series of issues on the Wikimedia side(large files, TIFF files, etc are all common things that contribute).

bettinacsmith commented 2 years ago

They are all JPEG files. Here is the portion of the batch that I was able to upload: https://commons.wikimedia.org/wiki/Category:Atil,_Syria