Some of the PGN files retrieved through /{id}/pgn seems to end prematurely

[Uglemat]

I made a script to download all my games and put it into a single PGN file to use them in the scid program, and I found out that some of PGN files seems to end prematurely. Of the ~900 games I downloaded, only about ~700 of them was imported by scid successfully, so it seems like it's a frequent problem.

Here are two URLs with that problem:

http://lichess.org/YTguq37D/pgn
http://lichess.org/O2KnrVnv/pgn

And here's what I get:

$ wget http://lichess.org/YTguq37D/pgn -qO -
[Event "Rated game"]
[Site "http://lichess.org/YTguq37D"]
[Date "2014.07.13"]
[White "FinkelFrand"]
[Black "Daaaaari"]
[Result "1-0"]
[WhiteElo "1622"]
[BlackElo "1506"]
[PlyCount "29"]
[Variant "Standard"]
[TimeControl "600+5"]
[ECO "A09"]
[Opening "Reti Opening, Advance Variation"]
[WhiteClock ":1:22"]
[BlackClock ":0:0"]
[Annotator "lichess.org"]

1. Nf3 d5 2. c4 d4 { Reti Opening, Advance Variation } 3. b3 c5 4. Bb2 Nc6 5. Na3 e5 6. d3 Nf6 7. e3 Bg4 8. Be2 Bxf3?! { (-0.44 → 0.06) Inaccuracy. The best move was Bd7. } (8... Bd7 9. O-O Bd6 10. Nb5 Be7 11. b4 a6 12. bxc5 dxe3 13. Nd6+ Bxd6 14. cxd6 exf2+ 15. Rxf2 Ng4 16. c5) 9. Bxf3 h5?! { (0.08 → 0.60) Inaccuracy. The best move was Be7. } (9... Be7 10. O-O O-O 11. Bxc6 bxc6 12. Nc2 Rb8 13. Re1 Re8 14. h3 Qd6 15. Qf3 Nd7 16. Rac1) 10. O-O g5?! { (0.61 → 1.47) Inaccuracy. The best move was Be7. } (10... Be7 11. Nc2 O-O 12. Bxh5 Nxh5 13. Qxh5 g6 14. Qh3 Bf6 15. Rae1 a6 16. Qf3 Rc8 17. Ba3 b6 18. exd4) 11. Nc2 g4?! { (1.47 → 2.45) Inaccuracy. The best move was Be7. } (11... Be7 12. Re1 O-O 13. Qd2 Nd7 14. Bxh5 a6 15. Bf3 Rb8) 12. Be2? { (2.45 → 0.46) Mistake. The best move was Bxc6+. } (12. Bxc6+ bxc6 13. f4 exf4 14. Rxf4 Bg7 15. exd4 O-O 16. dxc5 Nd7 17. Bxg7 Kxg7 18. Rf5 Rh8 19. Qd2 f6) 12... Rg8 13. exd4 cxd4?! { (0.42 → 1.15) Inaccuracy. The best move was exd4. } (13... exd4 14. Re1 Be7 15. b4 cxb4 16. Bf1 h4 17. Qe2 Kf8 18. Bxd4 Nxd4 19. Nxd4 Qxd4 20. Qxe7+ Kg7 21. g3) 14. Re1?! { (1.15 → 0.21) Inaccuracy. The best move was f4. } (14. f4 Bd6 15. Qe1 Qc7 16. Qh4 Be7 17. g3 O-O-O 18. Rae1 Qb6) 14... h4?! { (0.21 → 0.75) Inaccuracy. The best move was Qc7. } (14... Qc7 15. a3 h4 16. b4 a6 17. f4 gxf3 18. Bxf3 h3 19. Nxd4 Nxd4 20. Bxd4) 15. g3 { Black forfei
$ wget http://lichess.org/O2KnrVnv/pgn -qO -
[Event "Rated game"]
[Site "http://lichess.org/O2KnrVnv"]
[Date "2014.07.10"]
[White "Corblanc"]
[Black "FinkelFrand"]
[Result "0-1"]
[WhiteElo "1616"]
[BlackElo "1571"]
[PlyCount "102"]
[Variant "Standard"]
[TimeControl "480+5"]
[ECO "B54"]
[Opening "Sicilian Defense, Modern Variations"]
[WhiteClock ":4:43"]
[BlackClock ":0:27"]
[Annotator "lichess.org"]

1. e4 c5 2. Nf3 d6 3. d4 cxd4 4. Nxd4 { Sicilian Defense, Modern Variations } Nc6 5. c3 Nf6 6. Bb5 Qc7 7. Qe2 e6 8. Bg5 Be7 9. O-O O-O 10. Nd2 a6 11. Ba4 b5 12. Nxc6 Qxc6 13. Bc2 Bb7 14. Nf3 Rfe8 15. Bxf6?! { (0.27 → -0.25) Inaccuracy. The best move was a4. } (15. a4 e5 16. axb5 Qxb5 17. Bd3 Qb6 18. Rfd1 h6 19. Be3 Qc7 20. Nd2 Rab8 21. f3 d5 22. Qf2 Bc6) 15... Bxf6 16. Rad1 Rad8 17. Nd4 Qc7 18. Qh5?! { (-0.08 → -0.82) Inaccuracy. The best move was Rfe1. } (18. Rfe1 Rc8 19. a3 Qc4 20. Qd2 Red8 21. Bb3 Qc5 22. g3 Be7 23. Qe2 d5 24. exd5 Bxd5 25. Bxd5 Qxd5) 18... g6 19. Qf3 Bg7 20. Qe3 Qc5 21. Rfe1 b4 22. Nb3 Qxe3 23. Rxe3 bxc3 24. bxc3 d5 25. e5 Rc8 26. Nd4 Rc4 27. Bb3 Rc5 28. Ba4 Rec8 29. Rc1? { (-1.09 → -2.83) Mistake. The best move was Ne2. } (29. Ne2 a5 30. f4 Ba6 31. Kf2 Rb8 32. Bb3 Bf8 33. Rb1 a4 34. Bc2 Rxb1 35. Bxb1 Bxe2 36. Kxe2 Rb5) 29... Bh6 30. Bd7? { (-2.81 → -4.77) Mistake. The best move was Rb1. } (30. Rb1 Bxe3 31. fxe3 Rc4 32. Bc2 Bc6 33. Ne2 Bb5 34. a4 Bxa4 35. Bd3 R4c6 36. Kf2 Bb5 37. Bxb5 axb5) 30... R8c7 31. Bxe6? { (-4.65 → -5.88) Mistake. The best move was Ba4. } (31. Ba4 Rc4 32. Bc2 Bxe3 33. fxe3 Rxc3 34. g4 a5 35. Rb1 Rxe3 36. Kf2 Ra3 37. Rb2 Rc4 38. Rxb7 Rxd4) 31... Bxe3 32. fxe3 fxe6 33. Nxe6 Rxc3 34. Rxc3 Rxc3 35. Kf2 Rc2+ 36. Kg3 Rxa2 37. Nd4? { (-6.98 → -8.33) Mistake. The best move was Nc5. } (37. Nc5 Bc8 38. Kf3 Kf7 39. h3 Rc2 40. e6+ Kf6 41. Nb3 Bxe6 42. Nd4 Rc3 43. Ne2 Ra3 44. g4 a5) 37... Ra3? { (-8.33 → -7.12) Mistake. The best move was Kf7. } (37... Kf7 38. e6+ Ke7 39. h4 a5 40. Kf4 Rxg2 41. Kf3 Rb2 42. Kf4 a4 43. Ke5) 38. Kf2?! { (-7.12 → -7.67) Inaccuracy. The best move was Kf4. } (38. Kf4 Kf7 39. Nf3 Ra2 40. Ng5+ Ke7 41. Nxh7 Rxg2 42. h3 Rf2+ 43. Kg5 Rf3 44. h4 Rxe3 45. Kxg6 d4) 38... Rd3?! { (-7.67 → -6.70) Inaccuracy. The best move was Kf7. } (38... Kf7 39. Kf3 Ra2 40. h4 a5 41. e6+ Ke7 42. h5 gxh5 43. Kf4) 39. g3? { (-6.70 → -7.83) Mistake. The best move was Ke2. } (39. Ke2 Rc3 40. Kd2 Ra3 41. Nc2 Ra4 42. Nd4 Kf7 43. Kc3 Ra2 44. Nc2 Ke6 45. Kb3 Ra5) 39... Kf8?! { (-7.83 → -6.89) Inaccuracy. The best move was Rd2+. } (39... Rd2+ 40. Ne2 a5 41. Ke1 Rd3 42. e4 Re3 43. Kd2 Rxe4 44. Nc3 Rxe5 45. Kd3 Kf7 46. g4 Kf6 47. h4) 40. Ke2 Rxd4? { (-7.19 → -5.21) Mistake. The best move was Ra3. } (40... Ra3 41. h3 a5 42. Kd2 Ke7 43. Nb5 Ra2+ 44. Kc3 Ke6 45. Kd4 Ra4+ 46. Kc5 Kxe5 47. g4) 41. exd4 Ke7 42. h3?? { (-5.80 → -12.34) Blunder. The best move was Kd3. } (42. Kd3 a5 43. Kc3 h5 44. Kb3 Bc6 45. Kc2 Bd7 46. Kc3 Ke6 47. Kc2 a4 48. Kc3 Kf5 49. Kb4 Ke4) 42... Ke6? { (-12.34 → -7.06) Mistake. The best move was g5. } (42... g5 43. Kf3 Bc8 44. e6) 43. g4? { (-7.06 → -52.11) Mistake. The best move was Kd3. } (43. Kd3 a5 44. Kc3 Bc8 45. h4 Kf5 46. Kb3 Ke4 47. Ka4 Kxd4 48. Kxa5 Kxe5 49. Kb4) 43... h6 44. h4 Bc6 45. Kd2 Bb5 46. Kc3 h5 47. gxh5 gxh5 48. Kb4 Kf5 49. Kc5 Ke4 50. e6 Be8 5
$ 

Both of them ends prematurely.

[ornicar]

Just a sidenote: you should use the domain en.lichess.org instead of lichess.org to avoid the redirect, and cut the number of requests by two. Now I'm investigating the issue.

[ornicar]

Worth noting that the program httpie gets the entire output. However curl and wget get the truncated one.

[Uglemat]

$ wget http://en.lichess.org/YTguq37D/pgn -SqO -
  HTTP/1.1 200 OK
  Server: nginx
  Date: Tue, 15 Jul 2014 21:47:43 GMT
  Content-Type: text/plain; charset=utf-8
  Content-Length: 1750
  Connection: keep-alive
  Vary: Accept-Encoding
  Content-Disposition: attachment; filename=lichess_pgn_2014.07.13_FinkelFrand_vs_Daaaaari.YTguq37D.pgn
  X-UA-Compatible: IE=Edge,chrome=1
[Event "Rated game"]
[Site "http://lichess.org/YTguq37D"]
[Date "2014.07.13"]
[White "FinkelFrand"]
[Black "Daaaaari"]
[Result "1-0"]
[WhiteElo "1622"]
[BlackElo "1506"]
[PlyCount "29"]
[Variant "Standard"]
[TimeControl "600+5"]
[ECO "A09"]
[Opening "Reti Opening, Advance Variation"]
[WhiteClock ":1:22"]
[BlackClock ":0:0"]
[Annotator "lichess.org"]

1. Nf3 d5 2. c4 d4 { Reti Opening, Advance Variation } 3. b3 c5 4. Bb2 Nc6 5. Na3 e5 6. d3 Nf6 7. e3 Bg4 8. Be2 Bxf3?! { (-0.44 → 0.06) Inaccuracy. The best move was Bd7. } (8... Bd7 9. O-O Bd6 10. Nb5 Be7 11. b4 a6 12. bxc5 dxe3 13. Nd6+ Bxd6 14. cxd6 exf2+ 15. Rxf2 Ng4 16. c5) 9. Bxf3 h5?! { (0.08 → 0.60) Inaccuracy. The best move was Be7. } (9... Be7 10. O-O O-O 11. Bxc6 bxc6 12. Nc2 Rb8 13. Re1 Re8 14. h3 Qd6 15. Qf3 Nd7 16. Rac1) 10. O-O g5?! { (0.61 → 1.47) Inaccuracy. The best move was Be7. } (10... Be7 11. Nc2 O-O 12. Bxh5 Nxh5 13. Qxh5 g6 14. Qh3 Bf6 15. Rae1 a6 16. Qf3 Rc8 17. Ba3 b6 18. exd4) 11. Nc2 g4?! { (1.47 → 2.45) Inaccuracy. The best move was Be7. } (11... Be7 12. Re1 O-O 13. Qd2 Nd7 14. Bxh5 a6 15. Bf3 Rb8) 12. Be2? { (2.45 → 0.46) Mistake. The best move was Bxc6+. } (12. Bxc6+ bxc6 13. f4 exf4 14. Rxf4 Bg7 15. exd4 O-O 16. dxc5 Nd7 17. Bxg7 Kxg7 18. Rf5 Rh8 19. Qd2 f6) 12... Rg8 13. exd4 cxd4?! { (0.42 → 1.15) Inaccuracy. The best move was exd4. } (13... exd4 14. Re1 Be7 15. b4 cxb4 16. Bf1 h4 17. Qe2 Kf8 18. Bxd4 Nxd4 19. Nxd4 Qxd4 20. Qxe7+ Kg7 21. g3) 14. Re1?! { (1.15 → 0.21) Inaccuracy. The best move was f4. } (14. f4 Bd6 15. Qe1 Qc7 16. Qh4 Be7 17. g3 O-O-O 18. Rae1 Qb6) 14... h4?! { (0.21 → 0.75) Inaccuracy. The best move was Qc7. } (14... Qc7 15. a3 h4 16. b4 a6 17. f4 gxf3 18. Bxf3 h3 19. Nxd4 Nxd4 20. Bxd4) 15. g3 { Black forfei

The truncated content seems to be exactly 1750 characters, which is what the Content-Length HTTP header also says. I'm guessing that's related to the cause. When I used httpie it didn't seem to list the Content-Length header.

[Uglemat]

Using the --ignore-length option for wget makes it download the whole thing

[ornicar]

Good catch. I'm fixing the content length on the server.

[ornicar]

Got it fixed, will deploy tonight. Also I'll be adding a download of all games as a single file PGN. Very soon.

[ornicar]

There you go @SmartViking. The download link on your user page now delivers a fat PGN file, that scid should be able to import.

[Uglemat]

Works great, thanks :)