tbrennanw0rk
commented
9 years ago bump, I have having the same issue under CentOS 7.1 using the latest source.
Closed nonoderclown closed 9 years ago
tbrennanw0rk
commented
9 years ago bump, I have having the same issue under CentOS 7.1 using the latest source.
mdomsch
commented
9 years ago ---------- Forwarded message ---------- From: Mike Rooney Date: Thu, Oct 22, 2015 at 1:30 PM Subject: [S3tools-bugs] issue with proxy + HTTPS To: "s3tools-bugs@lists.sourceforge.net" s3tools-bugs@lists.sourceforge.net
Hey there! We are getting the following exception with a proxy configured:
$ s3cmd ls
Invoked as: /usr/local/Cellar/s3cmd/1.6.0/libexec/bin/s3cmd ls Problem: UnboundLocalError: local variable 'conn' referenced before assignment S3cmd: 1.6.0 python: 2.7.10 (default, Jul 14 2015, 19:46:27) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] environment LANG=en_US.UTF-8
Traceback (most recent call last):
File "/usr/local/Cellar/s3cmd/1.6.0/libexec/bin/s3cmd", line 2805, in
If I pull the connection creation line (conn = ConnMan.get(self.get_hostname(resource['bucket']))) above the try, I then get this:
Invoked as: /usr/local/Cellar/s3cmd/1.6.0/libexec/bin/s3cmd ls Problem: CertificateError: hostname 'proxy-developer.example.com' doesn't match 's3.amazonaws.com' S3cmd: 1.6.0 python: 2.7.10 (default, Jul 14 2015, 19:46:27) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] environment LANG=en_US.UTF-8
Traceback (most recent call last):
File "/usr/local/Cellar/s3cmd/1.6.0/libexec/bin/s3cmd", line 2805, in
So, perhaps this is valid and going to happen when using a proxy + HTTPS, I’m not sure? Either way, the original exception handling at S3.py:1084 (conn.counter = ConnMan.conn_max_counter) is assuming that the connection was successfully created in the first line of the try, which wasn’t the case, hence the “conn referenced before assignment”. Improved exception handling here would help to handle this case, and even a helpful error to suggest disabling HTTPS / using —no-check-certificate when using a proxy + HTTPS. Any thoughts?
mdomsch
commented
9 years ago Try with branch bug/598 now. I think I have this fixed.
mdomsch
commented
9 years ago 
fviard
commented
9 years ago Fix is now in master.
mdomsch
commented
9 years ago I hit a failure last night related to dns named buckets when using this fix and no proxy server. Need to investigate further. On Oct 26, 2015 3:50 PM, "Florent Viard" notifications@github.com wrote:
Closed #598 https://github.com/s3tools/s3cmd/issues/598.
— Reply to this email directly or view it on GitHub https://github.com/s3tools/s3cmd/issues/598#event-446052120.
s3cmd is a great tool. Thanks for the development and support ! I use s3cmd on a Windows machine which is behind a proxy. It works to connect to AWS (command "s3cmd ls") by using http but it fails with https.
Error message: WARNING: Retrying failed request: / (hostname '192.168.0.1' doesn't match 's3.amazonaws.com')
Notes: My proxy is 192.168.0.1 (Port 8888) s3cmd version: 1.5.2 Python version: 1.7.10 Note: here is no bucket name containing a dot (.)
On my machine I installed Cloudberry Explorer for Amazon S3 and could connect with http AND https. I tested a connection with s3cmd from another machine which is not behind a proxy: it works also fine with http and https.
So I guess that there are 2 possibilities:
here is the debug report with set use_https = True
C:\Python27\python c:\Python27\Scripts\s3cmd -s -d ls DEBUG: ConfigParser: Reading file 'C:\Users\siemens\AppData\Roaming\s3cmd.ini' DEBUG: ConfigParser: access_key->AK...17_chars...Q DEBUG: ConfigParser: access_token-> DEBUG: ConfigParser: add_encoding_exts-> DEBUG: ConfigParser: add_headers-> DEBUG: ConfigParser: bucket_location->EU DEBUG: ConfigParser: ca_certs_file-> DEBUG: ConfigParser: cache_file-> DEBUG: ConfigParser: check_ssl_certificate->False DEBUG: ConfigParser: cloudfront_host->s3.amazonaws.com DEBUG: ConfigParser: default_mime_type->binary/octet-stream DEBUG: ConfigParser: delay_updates->False DEBUG: ConfigParser: delete_after->False DEBUG: ConfigParser: delete_after_fetch->False DEBUG: ConfigParser: delete_removed->False DEBUG: ConfigParser: dry_run->False DEBUG: ConfigParser: enable_multipart->True DEBUG: ConfigParser: encoding->cp1252 DEBUG: ConfigParser: encrypt->False DEBUG: ConfigParser: expiry_date-> DEBUG: ConfigParser: expiry_days-> DEBUG: ConfigParser: expiry_prefix-> DEBUG: ConfigParser: follow_symlinks->False DEBUG: ConfigParser: force->False DEBUG: ConfigParser: get_continue->False DEBUG: ConfigParser: gpg_command->C:\Program Files (x86)\GNU\GnuPG\pub\gpg.exe DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s DEBUG: ConfigParser: gpg_passphrase->...-3_chars... DEBUG: ConfigParser: guess_mime_type->True DEBUG: ConfigParser: host_base->s3.amazonaws.com DEBUG: ConfigParser: host_bucket->%(bucket)s.s3.amazonaws.com DEBUG: ConfigParser: human_readable_sizes->False DEBUG: ConfigParser: ignore_failed_copy->False DEBUG: ConfigParser: invalidate_default_index_on_cf->False DEBUG: ConfigParser: invalidate_default_index_root_on_cf->True DEBUG: ConfigParser: invalidate_on_cf->False DEBUG: ConfigParser: list_md5->False DEBUG: ConfigParser: log_target_prefix-> DEBUG: ConfigParser: max_delete->-1 DEBUG: ConfigParser: mime_type-> DEBUG: ConfigParser: multipart_chunk_size_mb->15 DEBUG: ConfigParser: preserve_attrs->False DEBUG: ConfigParser: progress_meter->False DEBUG: ConfigParser: proxy_host->192.168.0.1 DEBUG: ConfigParser: proxy_port->8888 DEBUG: ConfigParser: put_continue->False DEBUG: ConfigParser: recursive->False DEBUG: ConfigParser: recv_chunk->4096 DEBUG: ConfigParser: reduced_redundancy->False DEBUG: ConfigParser: restore_days->1 DEBUG: ConfigParser: secret_key->nz...37_chars...w DEBUG: ConfigParser: send_chunk->4096 DEBUG: ConfigParser: server_side_encryption->False DEBUG: ConfigParser: signature_v2->False DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com DEBUG: ConfigParser: skip_existing->False DEBUG: ConfigParser: socket_timeout->300 DEBUG: ConfigParser: urlencoding_mode->normal DEBUG: ConfigParser: use_https->False DEBUG: ConfigParser: use_mime_magic->True DEBUG: ConfigParser: verbosity->WARNING DEBUG: ConfigParser: website_endpoint->http://%(bucket)s.s3-website-%(location)s.amazonaws.com/ DEBUG: ConfigParser: website_error-> DEBUG: ConfigParser: website_index->index.html ERROR: Option --progress is not yet supported on MS Windows platform. Assuming --no-progress. DEBUG: Updating Config.Config cache_file -> DEBUG: Updating Config.Config follow_symlinks -> False DEBUG: Updating Config.Config use_https -> True DEBUG: Updating Config.Config verbosity -> 10 DEBUG: Unicodising 'ls' using cp1252 DEBUG: Command: ls DEBUG: CreateRequest: resource[uri]=/ DEBUG: Using signature v2 DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Thu, 06 Aug 2015 06:53:52 +0000\n/' DEBUG: Processing request, please wait... DEBUG: get_hostname(None): s3.amazonaws.com DEBUG: ConnMan.get(): creating new connection: proxy://192.168.0.1:8888 DEBUG: Using ca_certs_file None DEBUG: Disabling hostname checking DEBUG: proxied HTTPSConnection(192.168.0.1, 8888) DEBUG: tunnel to s3.amazonaws.com WARNING: Retrying failed request: / (hostname '192.168.0.1' doesn't match 's3.amazonaws.com') WARNING: Waiting 3 sec... DEBUG: Using signature v2 DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Thu, 06 Aug 2015 06:53:55 +0000\n/' DEBUG: Processing request, please wait... DEBUG: get_hostname(None): s3.amazonaws.com DEBUG: ConnMan.get(): creating new connection: proxy://192.168.0.1:8888 DEBUG: proxied HTTPSConnection(192.168.0.1, 8888) DEBUG: tunnel to s3.amazonaws.com WARNING: Retrying failed request: / (hostname '192.168.0.1' doesn't match 's3.amazonaws.com') WARNING: Waiting 6 sec...
please find my s3cmd.ini file where I remove access key and secret key (***)
[default] access_key = ** access_token = add_encoding_exts = add_headers = bucket_location = EU ca_certs_file = cache_file = check_ssl_certificate = False cloudfront_host = s3.amazonaws.com default_mime_type = binary/octet-stream delay_updates = False delete_after = False delete_after_fetch = False delete_removed = False dry_run = False enable_multipart = True encoding = cp1252 encrypt = False expiry_date = expiry_days = expiry_prefix = follow_symlinks = False force = False get_continue = False gpg_command = C:\Program Files (x86)\GNU\GnuPG\pub\gpg.exe gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_passphrase = guess_mime_type = True host_base = s3.amazonaws.com host_bucket = %(bucket)s.s3.amazonaws.com human_readable_sizes = False ignore_failed_copy = False invalidate_default_index_on_cf = False invalidate_default_index_root_on_cf = True invalidate_on_cf = False list_md5 = False log_target_prefix = max_delete = -1 mime_type = multipart_chunk_size_mb = 15 preserve_attrs = False progress_meter = False proxy_host = 192.168.0.1 proxy_port = 8888 put_continue = False recursive = False recv_chunk = 4096 reduced_redundancy = False restore_days = 1 secret_key = send_chunk = 4096 server_side_encryption = False signature_v2 = False simpledb_host = sdb.amazonaws.com skip_existing = False socket_timeout = 300 urlencoding_mode = normal use_https = True use_mime_magic = True verbosity = WARNING website_endpoint = http://%(bucket)s.s3-website-%(location)s.amazonaws.com/ website_error = website_index = index.html
Thanks for your help !