search

GoogleCloudPlatform / gsutil

A command line tool for interacting with cloud storage services.
Apache License 2.0
876 stars 335 forks source link

gsutil hangs #293

Closed bakks closed 8 years ago

bakks commented 9 years ago

gsutil sometimes hangs for us, we haven't been able to figure out a root cause. This is a copy of: https://code.google.com/p/google-cloud-sdk/issues/detail?id=204

What steps will reproduce the problem? Only happens intermittently during a 'gsutil cp' command, no clear conditions which trigger it.

What is the expected output? What do you see instead? I expect the upload to eventually complete. Instead the command stops outputting progress updates and just hangs (in this case, for over 2 days)

What is the output of 'gcloud info'?

Google Cloud SDK [0.9.71]

Platform: [Linux, x86_64]
Python Version: [2.7.5 (default, Jun 24 2015, 00:41:19)  [GCC 4.8.3 20140911 (Red Hat 4.8.3-9)]]
Site Packages: [Disabled]

Installation Root: [/usr/local/share/google/google-cloud-sdk]
Installed Components:
  core: [2015.07.24]
  core-nix: [2015.07.02]
  gcloud: [2015.07.24]
  gsutil-nix: [4.13]
  gsutil: [4.13]
  bq: [2.0.18]
  bq-nix: [2.0.18]
System PATH: [/usr/local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/rvm/bin]
Cloud SDK on PATH: [False]

Installation Properties: [/usr/local/share/google/google-cloud-sdk/properties]
User Config Directory: [/home/stephen/.config/gcloud]
User Properties: [/home/stephen/.config/gcloud/properties]
Current Workspace: [None]
Workspace Config Directory: [None]
Workspace Properties: [None]

Account: [<SERVICE_ACCOUNT_EMAIL>]
Project: [<PROJECT_ID>]

Current Properties:
  [core]
    project: [<PROJECT_ID>]
    account: [<SERVICE_ACCOUNT_EMAIL>]

Logs Directory: [/home/stephen/.config/gcloud/logs]
Last Log File: [/home/stephen/.config/gcloud/logs/2015.08.14/15.59.27.860520.log]

Please provide any additional information below.

Symptoms are similar to this issue: https://code.google.com/p/google-cloud-sdk/issues/detail?id=166, but with gsutil instead of gcloud

Ran pstree and strace as requested in the other ticket, there were quite a number of subprocesses so I only looked at some of them. Output:

$ pstree -apl 31253
gsutil,31253 /usr/local/bin/gsutil -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
  └─python,31266 /usr/local/bin/../share/google/google-cloud-sdk/bin/bootstrapping/gsutil.py -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
      └─python,31273 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          ├─python,31282 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31335
          │   ├─{python},31553
          │   ├─{python},31785
          │   ├─{python},31804
          │   ├─{python},31809
          │   ├─{python},31819
          │   ├─{python},31821
          │   ├─{python},31824
          │   ├─{python},31825
          │   ├─{python},31829
          │   ├─{python},31830
          │   ├─{python},31833
          │   ├─{python},31835
          │   ├─{python},31836
          │   ├─{python},31840
          │   ├─{python},31844
          │   ├─{python},31849
          │   ├─{python},31853
          │   ├─{python},31855
          │   ├─{python},31856
          │   ├─{python},31861
          │   ├─{python},31862
          │   ├─{python},31863
          │   ├─{python},31864
          │   ├─{python},31865
          │   ├─{python},31866
          │   ├─{python},31867
          │   ├─{python},31868
          │   ├─{python},31869
          │   ├─{python},31871
          │   ├─{python},31883
          │   ├─{python},31884
          │   ├─{python},31885
          │   └─{python},31886
          ├─python,31283 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─python,31340 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31339
          │   ├─{python},31442
          │   ├─{python},31677
          │   ├─{python},31681
          │   ├─{python},31690
          │   ├─{python},31703
          │   ├─{python},31708
          │   ├─{python},31715
          │   ├─{python},31721
          │   ├─{python},31738
          │   ├─{python},31757
          │   ├─{python},31781
          │   ├─{python},31782
          │   ├─{python},31783
          │   ├─{python},31790
          │   ├─{python},31791
          │   ├─{python},31799
          │   ├─{python},31805
          │   ├─{python},31807
          │   ├─{python},31810
          │   ├─{python},31813
          │   ├─{python},31817
          │   ├─{python},31823
          │   ├─{python},31832
          │   ├─{python},31837
          │   ├─{python},31842
          │   ├─{python},31846
          │   ├─{python},31851
          │   ├─{python},31854
          │   ├─{python},31858
          │   ├─{python},31873
          │   ├─{python},31875
          │   ├─{python},31877
          │   └─{python},31880
          ├─python,31317 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          ├─python,31345 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31421
          │   ├─{python},31422
          │   ├─{python},31423
          │   ├─{python},31425
          │   ├─{python},31426
          │   ├─{python},31427
          │   ├─{python},31428
          │   ├─{python},31430
          │   ├─{python},31431
          │   ├─{python},31433
          │   └─{python},31581
          ├─python,31346 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31446
          │   ├─{python},31447
          │   ├─{python},31448
          │   ├─{python},31449
          │   ├─{python},31450
          │   ├─{python},31452
          │   ├─{python},31453
          │   ├─{python},31456
          │   ├─{python},31457
          │   └─{python},31459
          ├─python,31348 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31479
          │   ├─{python},31482
          │   ├─{python},31483
          │   ├─{python},31484
          │   ├─{python},31486
          │   ├─{python},31489
          │   ├─{python},31490
          │   ├─{python},31491
          │   ├─{python},31493
          │   └─{python},31494
          ├─python,31351 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31498
          │   ├─{python},31499
          │   ├─{python},31501
          │   ├─{python},31502
          │   ├─{python},31505
          │   ├─{python},31506
          │   ├─{python},31509
          │   ├─{python},31512
          │   ├─{python},31516
          │   └─{python},31523
          ├─python,31354 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31515
          │   ├─{python},31519
          │   ├─{python},31522
          │   ├─{python},31526
          │   ├─{python},31529
          │   ├─{python},31533
          │   ├─{python},31535
          │   ├─{python},31537
          │   ├─{python},31541
          │   └─{python},31546
          ├─python,31357 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31511
          │   ├─{python},31514
          │   ├─{python},31517
          │   ├─{python},31520
          │   ├─{python},31525
          │   ├─{python},31528
          │   ├─{python},31532
          │   ├─{python},31539
          │   ├─{python},31542
          │   └─{python},31545
          ├─python,31360 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31504
          │   ├─{python},31507
          │   ├─{python},31508
          │   ├─{python},31510
          │   ├─{python},31513
          │   ├─{python},31518
          │   ├─{python},31521
          │   ├─{python},31524
          │   ├─{python},31530
          │   └─{python},31534
          ├─python,31363 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31527
          │   ├─{python},31531
          │   ├─{python},31536
          │   ├─{python},31538
          │   ├─{python},31540
          │   ├─{python},31543
          │   ├─{python},31544
          │   ├─{python},31547
          │   ├─{python},31548
          │   └─{python},31549
          ├─python,31555 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31649
          │   ├─{python},31651
          │   ├─{python},31656
          │   ├─{python},31658
          │   ├─{python},31660
          │   ├─{python},31662
          │   ├─{python},31663
          │   ├─{python},31667
          │   ├─{python},31668
          │   └─{python},31669
          ├─python,31556 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31637
          │   ├─{python},31638
          │   ├─{python},31643
          │   ├─{python},31644
          │   ├─{python},31646
          │   ├─{python},31648
          │   ├─{python},31650
          │   ├─{python},31653
          │   ├─{python},31654
          │   └─{python},31657
          ├─python,31558 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31684
          │   ├─{python},31686
          │   ├─{python},31691
          │   ├─{python},31694
          │   ├─{python},31695
          │   ├─{python},31697
          │   ├─{python},31699
          │   ├─{python},31701
          │   ├─{python},31704
          │   └─{python},31705
          ├─python,31561 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31709
          │   ├─{python},31713
          │   ├─{python},31716
          │   ├─{python},31718
          │   ├─{python},31719
          │   ├─{python},31723
          │   ├─{python},31725
          │   ├─{python},31726
          │   ├─{python},31727
          │   └─{python},31729
          ├─python,31565 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31751
          │   ├─{python},31754
          │   ├─{python},31755
          │   ├─{python},31756
          │   ├─{python},31758
          │   ├─{python},31760
          │   ├─{python},31763
          │   ├─{python},31765
          │   ├─{python},31769
          │   └─{python},31772
          ├─python,31568 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31732
          │   ├─{python},31736
          │   ├─{python},31739
          │   ├─{python},31740
          │   ├─{python},31742
          │   ├─{python},31743
          │   ├─{python},31744
          │   ├─{python},31746
          │   ├─{python},31747
          │   └─{python},31748
          ├─python,31573 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31762
          │   ├─{python},31764
          │   ├─{python},31766
          │   ├─{python},31768
          │   ├─{python},31771
          │   ├─{python},31773
          │   ├─{python},31774
          │   ├─{python},31776
          │   ├─{python},31777
          │   └─{python},31779
          ├─python,31575 /usr/local/share/google/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<PROJECT_ID> -o GoogleCompute:service_account=<SERVICE_ACCOUNT> -m -o GSUtil:parallel_composite_upload_threshold=150M cp <LOCAL_FILE_PATH> gs://<GCS_FILE_PATH>
          │   ├─{python},31786
          │   ├─{python},31789
          │   ├─{python},31793
          │   ├─{python},31794
          │   ├─{python},31796
          │   ├─{python},31797
          │   ├─{python},31798
          │   ├─{python},31801
          │   ├─{python},31803
          │   └─{python},31806
          └─{python},31377

$ sudo strace -f -p 31253
Process 31253 attached
wait4(-1,

$ sudo strace -f -p 31266
Process 31266 attached
wait4(31273,

$ sudo strace -f -p 31273
Process 31273 attached with 2 threads
[pid 31273] read(6,  <unfinished ...>
[pid 31377] futex(0x7fe96c001120, FUTEX_WAIT_PRIVATE, 0, NULL

$ sudo strace -f -p 31282
Process 31282 attached with 35 threads
[pid 31886] read(40,  <unfinished ...>
[pid 31885] read(38,  <unfinished ...>
[pid 31884] read(36,  <unfinished ...>
[pid 31883] read(35,  <unfinished ...>
[pid 31871] read(37,  <unfinished ...>
[pid 31869] read(34,  <unfinished ...>
[pid 31868] read(32,  <unfinished ...>
[pid 31867] read(33,  <unfinished ...>
[pid 31866] read(31,  <unfinished ...>
[pid 31865] read(30,  <unfinished ...>
[pid 31864] read(29,  <unfinished ...>
[pid 31863] read(26,  <unfinished ...>
[pid 31862] read(27,  <unfinished ...>
[pid 31861] read(21,  <unfinished ...>
[pid 31856] read(28,  <unfinished ...>
[pid 31855] read(24,  <unfinished ...>
[pid 31853] read(25,  <unfinished ...>
[pid 31849] read(10,  <unfinished ...>
[pid 31844] read(20,  <unfinished ...>
[pid 31840] read(23,  <unfinished ...>
[pid 31836] read(22,  <unfinished ...>
[pid 31835] read(16,  <unfinished ...>
[pid 31833] read(18,  <unfinished ...>
[pid 31830] read(19,  <unfinished ...>
[pid 31829] read(17,  <unfinished ...>
[pid 31825] read(14,  <unfinished ...>
[pid 31824] read(13,  <unfinished ...>
[pid 31821] read(8,  <unfinished ...>
[pid 31819] read(15,  <unfinished ...>
[pid 31809] read(12,  <unfinished ...>
[pid 31804] read(11,  <unfinished ...>
[pid 31785] read(9,  <unfinished ...>
[pid 31553] read(7,  <unfinished ...>
[pid 31335] read(4,  <unfinished ...>
[pid 31282] accept(6,

$ sudo strace -f -p 31283
Process 31283 attached with 35 threads
[pid 31880] read(37,  <unfinished ...>
[pid 31877] read(39,  <unfinished ...>
[pid 31875] read(38,  <unfinished ...>
[pid 31873] read(35,  <unfinished ...>
[pid 31858] read(36,  <unfinished ...>
[pid 31854] read(34,  <unfinished ...>
[pid 31851] read(33,  <unfinished ...>
[pid 31846] read(31,  <unfinished ...>
[pid 31842] read(32,  <unfinished ...>
[pid 31837] read(30,  <unfinished ...>
[pid 31832] read(20,  <unfinished ...>
[pid 31823] read(29,  <unfinished ...>
[pid 31817] read(28,  <unfinished ...>
[pid 31813] read(27,  <unfinished ...>
[pid 31810] read(26,  <unfinished ...>
[pid 31807] read(25,  <unfinished ...>
[pid 31805] read(24,  <unfinished ...>
[pid 31799] read(23,  <unfinished ...>
[pid 31791] read(21,  <unfinished ...>
[pid 31790] read(22,  <unfinished ...>
[pid 31783] read(19,  <unfinished ...>
[pid 31782] read(17,  <unfinished ...>
[pid 31781] read(18,  <unfinished ...>
[pid 31757] read(12,  <unfinished ...>
[pid 31738] read(16,  <unfinished ...>
[pid 31721] read(7,  <unfinished ...>
[pid 31715] read(15,  <unfinished ...>
[pid 31708] read(8,  <unfinished ...>
[pid 31703] read(11,  <unfinished ...>
[pid 31690] read(14,  <unfinished ...>
[pid 31681] read(9,  <unfinished ...>
[pid 31677] read(10,  <unfinished ...>
[pid 31442] futex(0x7fe964001170, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 31339] futex(0x7fe974019f10, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 31283] accept(6,

$ sudo strace -f -p 31317
Process 31317 attached
accept(8,

$ sudo strace -f -p 31346
Process 31346 attached with 11 threads
[pid 31459] futex(0x7fe948001300, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 31457] futex(0x7fe954001300, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 31456] futex(0x7fe950001300, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 31453] futex(0x7fe95c001300, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 31452] futex(0x7fe958001300, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 31450] futex(0x7fe964001610, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 31449] futex(0x7fe960001300, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 31448] futex(0x7fe968001130, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 31447] futex(0x7fe96c001300, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 31346] futex(0x7fe98717b000, FUTEX_WAIT, 0, NULL <unfinished ...>
[pid 31446] futex(0x7fe9740014d0, FUTEX_WAIT_PRIVATE, 0, NULL
thobrla commented 9 years ago

Could you send an email to gs-team@google.com and attach the stderr/stdout of gsutil -D your-command ... for the hanging case?

smreichling commented 9 years ago

Sorry for the delay, we just had another occurrence of this with debug logging turned on. I've emailed the debug log output to you.

aquarapid commented 8 years ago

Any update on this; we are seeing it rather frequently. For us it happens within minutes when using trickle to limit upload speed of a gsutil rsync. Without trickle, it almost never happens. This is problematic, since there isn't really another way than trickle throttle uploads...

thobrla commented 8 years ago

Per private conversation with @smreichling around Dec 2015, this hang no longer reproduces, so I'm closing this issue.

@aquarapid - could you open a separate issue on the trickle problems you're seeing (including the commands you're using)?