search

bozaro / git-lfs-migrate

Simple project for convert old repository for using git-lfs feature
MIT License
222 stars 29 forks source link

Multi thread support causes problems with github lfs #3

Closed CPlusPlus17 closed 9 years ago

CPlusPlus17 commented 9 years ago

Hello again! :)

Since multi thread support is added I'm getting:

Exception in thread "main" java.io.IOException: Error writing request body to server
    at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3478)
    at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3461)
    at git.lfs.migrate.GitConverter.upload(GitConverter.java:320)
    at git.lfs.migrate.GitConverter.access$300(GitConverter.java:26)
    at git.lfs.migrate.GitConverter$5.convert(GitConverter.java:254)
    at git.lfs.migrate.Main.lambda$processMultipleThreads$0(Main.java:123)
    at git.lfs.migrate.Main$$Lambda$1/1059072014.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    Suppressed: java.io.IOException: insufficient data written
        at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.close(HttpURLConnection.java:3500)
        at git.lfs.migrate.GitConverter.upload(GitConverter.java:322)
        ... 9 more

and

Exception in thread "main" java.net.SocketException: Unexpected end of file from server
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:792)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1535)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
    at git.lfs.migrate.GitConverter.upload(GitConverter.java:323)
    at git.lfs.migrate.GitConverter.access$300(GitConverter.java:26)
    at git.lfs.migrate.GitConverter$5.convert(GitConverter.java:254)
    at git.lfs.migrate.Main.lambda$processMultipleThreads$0(Main.java:123)
    at git.lfs.migrate.Main$$Lambda$1/1059072014.run(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

I thinks this has something to do with API limitation on github or S3. The error occurs at random, sometimes 100 files can be uploaded, sometimes some thousands.

I would suggest to reduce it to 4 threads as default, seems to run stable here. (currently testing with a 250Mbit/s upload connection)

Greets Manuel

CPlusPlus17 commented 9 years ago

Too the counting seems a bit weird since the multithread change:

[pool-1-thread-3] INFO git.lfs.migrate.Main -   completed: 593/26596
[pool-1-thread-4] INFO git.lfs.migrate.Main -   completed: 596/26596
[pool-1-thread-2] INFO git.lfs.migrate.Main -   completed: 597/26596
[pool-1-thread-3] INFO git.lfs.migrate.Main -   completed: 598/26596
[pool-1-thread-1] INFO git.lfs.migrate.Main -   completed: 601/26596

I suggest the split the threads on FIFO and not static. Currently it seems you just split the file amount to the threads. This can in the worst case cause a thread with only big files and other threads with only small files.

In my opinion you should create one big queue and after a thread is done, it can get an entry from it. So you can get best performance and can return to normal global counting.

bozaro commented 9 years ago

How much LFS-objects in your repository in your estimation?

bozaro commented 9 years ago

I added more detail error messages. Can you find out what the error occurs when uploading the file?

bozaro commented 9 years ago

As workaround you also can limit thread count by --threads 1 (default value: current CPU cores).

bozaro commented 9 years ago

I do not think that the problem is directly related to multithreading. Most likely it provokes some limits S3 or GitHub. In our local environment (git-as-svn as LFS server) converting the repository (~80Gb bare repository after git gc) passed without any problems. The load on the network is ~400 megabits.

CPlusPlus17 commented 9 years ago

I'm currently testing with 16 threads with our newest commit.

Yes, I'm sure there is some problem on github or s3. I have a hard time to get the things pushed, it's quitting with all sort of erros. (git push origin master)

Somehow it seems it's depending on the day time and the load on github/s3.

@Object count: [main] INFO git.lfs.migrate.Main - Found objects: 33808

CPlusPlus17 commented 9 years ago

Today everything worked with 16 threads without any problem. I wait some hour for the next test until USA gets more active. (I'm in Europa)

bozaro commented 9 years ago

I add code for trying to fulfill each request 3 times before the error. I think it will survive the occasional network failure.

CPlusPlus17 commented 9 years ago

I did the run three more times, worked without problems.

But I still can't push to github:

java -jar newest_debug.jar -s /home/mgysin/Projects/migration/01_Game.git -d /home/mgysin/Projects/migration/ROAC -l https://xxx:yyy@api.github.com/lfs/QuantumBytesInc/ROAC/ .0 .4 .a .backup .bmp .chm .dat .dds .dle .dli .dll .dlu .dmg .dylib .exe .fbx .fla .flv .gif .ico .jpg .lib .mesh .mll .mp3 .mpk .o .oga .ogv .old .pak .pdb .pem .pfx .pkf .png .rar .sanim .sdf .smesh .so .suo .swf .ter .ter000* .tga .ttf .txt .ung .wav .woff .zip .bin .core -t 16
[main] INFO git.lfs.migrate.Main -   completed: 26596/26596
[main] INFO git.lfs.migrate.Main - Converting graph in single thread...
[main] INFO git.lfs.migrate.Main -   completed: 1/7212
[main] INFO git.lfs.migrate.Main -   completed: 5688/7212
[main] INFO git.lfs.migrate.Main -   completed: 7212/7212
[main] INFO git.lfs.migrate.Main - Recreating refs...
[main] INFO git.lfs.migrate.Main -   convert ref: 50208d65a4d0aca9ef1a05d15afb7467b4254390 -> 3eb3a45d112c040cc3764a80b875e7abe37ae736 (HEAD)
[main] INFO git.lfs.migrate.Main -   convert ref: eafb9b8b4fd1a106ff9bb2e7b697f44b81e5a7a6 -> 30a96d899943ba6a9c153740f32418104484a293 (refs/heads/chatbugfix)
[main] INFO git.lfs.migrate.Main -   convert ref: 402bd1a26ce0485b3a8ffc5a398a74cc8ab9bd4c -> b38a50451eeb1a41fb71ff105bba4c8d38f463a2 (refs/heads/coding)
[main] INFO git.lfs.migrate.Main -   convert ref: a4425df8143f2ad7932c1aa1ae501f236c60fad8 -> 7ca0de0ab46055ce5a0eb9a44296ca849c21fdc2 (refs/heads/coherent)
[main] INFO git.lfs.migrate.Main -   convert ref: 08d0bb0197d693c94609e3ab6f1b6812a058a3ed -> de1c12999800bc8076fd2514be66c4622b5dfce6 (refs/heads/graphic)
[main] INFO git.lfs.migrate.Main -   convert ref: e2490162ee28cad73e8671f6a5d731c135c50de5 -> df7ef7d0723182219940cd201d7c44f285326f81 (refs/heads/graphic__gathering_crafting_loginscreen)
[main] INFO git.lfs.migrate.Main -   convert ref: 990944c90b4a8aae3e08878b1273d157cea45a9d -> fe4b2d051a5ed6598fb95acbb8fb80a2a7d00197 (refs/heads/graphic_speedtree)
[main] INFO git.lfs.migrate.Main -   convert ref: 50208d65a4d0aca9ef1a05d15afb7467b4254390 -> 3eb3a45d112c040cc3764a80b875e7abe37ae736 (refs/heads/master)
[main] INFO git.lfs.migrate.Main -   convert ref: 684d94e852e1eea12f8dc52fc0fa3d81cd37f755 -> 65663300aa7d505e7fe7d5b623c07b785634d5b7 (refs/heads/new_gui)
[main] INFO git.lfs.migrate.Main -   convert ref: 98a617d8b9f65eab2a17a41b3ed5c195841a964d -> 6347bc69be3fbe7afe8d34d89acff493e2cbfa5c (refs/heads/prerelease)
[main] INFO git.lfs.migrate.Main -   convert ref: 92c7feef408b5c687d14269985f2df3fad5336c9 -> d0b9a3d6677220593e12beaf10d5c781fde011d0 (refs/heads/server_temp)
[main] INFO git.lfs.migrate.Main -   convert ref: 695b16d4ae84e4d36274bb50199028dba9a9638c -> a114cdc7135f83aee2f50922a60efda120544a73 (refs/heads/temp)
[main] INFO git.lfs.migrate.Main -   convert ref: 695b16d4ae84e4d36274bb50199028dba9a9638c -> a114cdc7135f83aee2f50922a60efda120544a73 (refs/heads/tracker)
[main] INFO git.lfs.migrate.Main -   convert ref: cf2a1d4ce48d9f5571c2d2ec1f15df8554e89770 -> 492bc3473cdcaca42745cc618bf9168e97c5664d (refs/heads/tracker_manager)
[main] INFO git.lfs.migrate.Main -   convert ref: 2437d54b7830c83cb3d0338af0c8ec54120c0af1 -> d4a154868e49ec76325ff2fab64bc685ce0f8fd3 (refs/tags/A5R)
[main] INFO git.lfs.migrate.Main -   convert ref: ec3e49204e4f86318c5213fadb7b1b52455ab1f0 -> f4e9c6073a504ffb5cd07e33a0ad15132ff8bb7d (refs/tags/A5RC1)
[main] INFO git.lfs.migrate.Main -   convert ref: faa4fc600349113a5ff9c699e4f04459d41c8e97 -> f67aa78fd3de1437fd8c2208386531367f44018b (refs/tags/A5RC2)
[main] INFO git.lfs.migrate.Main -   convert ref: bdc67f81d37831df3451fca5c496202e8f7b7403 -> a8005f0521489c6025ad2110a71623613fc00588 (refs/tags/A5RC3)
[main] INFO git.lfs.migrate.Main -   convert ref: d6ac735f87436e5616a136519bda8befc01006f8 -> 36623b6b12c6ab79334eabdad8b90c731d9a3875 (refs/tags/A5RC4)
[main] INFO git.lfs.migrate.Main -   convert ref: bcd6b6c8782dda5dc4b043cdccb1e0e165016100 -> a9ee876e9e63bb0b29c6e9960706ad8afc76f0ad (refs/tags/A5RC5)
[main] INFO git.lfs.migrate.Main -   convert ref: 49d3f04b0a800f2e2add627a2b97377e9767bf56 -> 8169fb11e61c0cb41ae7add5f1b6c922ae3d9547 (refs/tags/R5)
[main] INFO git.lfs.migrate.Main - Convert time: 638379
java -jar newest_debug.jar -s /home/mgysin/Projects/migration/01_Game.git -d   388.88s user 15.54s system 63% cpu 10:38.64 total
git fsck && git push --mirror git@github.com:QuantumBytesInc/ROAC.git
git fsck && git push --mirror git@github.com:QuantumBytesInc/ROAC.git
Prüfe Objekt-Verzeichnisse: 100% (256/256), Fertig.
git fsck  23.13s user 1.09s system 99% cpu 24.357 total
Zähle Objekte: 33808, Fertig.
Delta compression using up to 16 threads.
Komprimiere Objekte: 100% (33172/33172), Fertig.
Schreibe Objekte: 100% (33808/33808), 1.36 GiB | 133.00 KiB/s, Fertig.
Total 33808 (delta 9621), reused 0 (delta 0)
remote: warning: File lib/host/CoherentUI_Host.bin is 69.62 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File lib/libUnigine_x86.so is 86.32 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File lib/libUnigine_x86.so is 81.48 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File lib/libUnigine_x86d.so is 95.44 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File lib/host/core.1504 is 90.98 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File lib/Unigine_x64d.pdb is 76.35 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File lib/Unigine_x86d.pdb is 71.49 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File lib/libUnigine_x86.so is 80.69 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File lib/libUnigine_x86d.so is 94.66 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: error: GH001: Large files detected.
remote: error: Trace: 5f0219c99dbefde83e1644e0d2fb3ed8
remote: error: See http://git.io/iEPt8g for more information.
remote: error: File lib/libUnigine_x64.so is 109.19 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File lib/libUnigine_x64d.so is 128.21 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File lib/libUnigine_x86d.so is 101.00 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File lib/libUnigine_x64.so is 103.48 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File lib/libUnigine_x64d.so is 121.67 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File lib/libUnigine_x64.so is 102.43 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File lib/libUnigine_x64d.so is 120.63 MB; this exceeds GitHub's file size limit of 100.00 MB
To git@github.com:QuantumBytesInc/ROAC.git
 ! [remote rejected] chatbugfix -> chatbugfix (pre-receive hook declined)
 ! [remote rejected] coding -> coding (pre-receive hook declined)
 ! [remote rejected] coherent -> coherent (pre-receive hook declined)
 ! [remote rejected] graphic -> graphic (pre-receive hook declined)
 ! [remote rejected] graphic__gathering_crafting_loginscreen -> graphic__gathering_crafting_loginscreen (pre-receive hook declined)
 ! [remote rejected] graphic_speedtree -> graphic_speedtree (pre-receive hook declined)
 ! [remote rejected] master -> master (pre-receive hook declined)
 ! [remote rejected] new_gui -> new_gui (pre-receive hook declined)
 ! [remote rejected] prerelease -> prerelease (pre-receive hook declined)
 ! [remote rejected] server_temp -> server_temp (pre-receive hook declined)
 ! [remote rejected] temp -> temp (pre-receive hook declined)
 ! [remote rejected] tracker -> tracker (pre-receive hook declined)
 ! [remote rejected] tracker_manager -> tracker_manager (pre-receive hook declined)
 ! [remote rejected] A5R -> A5R (pre-receive hook declined)
 ! [remote rejected] A5RC1 -> A5RC1 (pre-receive hook declined)
 ! [remote rejected] A5RC2 -> A5RC2 (pre-receive hook declined)
 ! [remote rejected] A5RC3 -> A5RC3 (pre-receive hook declined)
 ! [remote rejected] A5RC4 -> A5RC4 (pre-receive hook declined)
 ! [remote rejected] A5RC5 -> A5RC5 (pre-receive hook declined)
 ! [remote rejected] R5 -> R5 (pre-receive hook declined)
error: Fehler beim Versenden einiger Referenzen nach 'git@github.com:QuantumBytesInc/ROAC.git'
git push --mirror git@github.com:QuantumBytesInc/ROAC.git  470.23s user 7.42s system 7% cpu 1:52:38.45 total

We have 30 .so files in the project, all of them get tracked by LFS but these on the top log are somehow ignored. The same situation is for the .pdb files, all but two get LFS flagged.

git lfs ls-files | grep .so
...
lib/libBoost_x64.so
lib/libCoherentUI.so
lib/libCoherentUIPlugin_x64.so
lib/libRenderer_x64.so
lib/libRenderer_x64d.so
lib/libRenderer_x86.so
lib/libRenderer_x86d.so
lib/libopenal.so
...
bozaro commented 9 years ago

I see source code, and looks like not converted files have "executable" flag in some revision.

I can fix this bug in some hours.

CPlusPlus17 commented 9 years ago

Thank you very much for your analysis. I wait for your commit. :)

bozaro commented 9 years ago

I fix this annoyance error: d2e19aa87cace25936becf4cbac15cc16ce4df79

CPlusPlus17 commented 9 years ago

Everything is now working! Thank you very much! :)

bozaro commented 9 years ago

Thank you for your patience :)