search

tredly / tredly-host

[DEPRECATED] Please use https://github.com/tredly/tredly instead
MIT License
21 stars 1 forks source link

nginx reverse proxy problem #36

Closed dinapappor closed 8 years ago

dinapappor commented 8 years ago

Actually I think this is an ipfw problem. Since today I can't send traffic to a container with nginx.

Nginx seems to to timeout to upstream (the container)

2016/05/13 14:11:28 [error] 79728#0: *22 upstream timed out (60: Operation timed out) while connecting to upstream, client: 172.16.0.151, server: sub.domain.tld, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.99.209.52:80/favicon.ico", host: "sub.domain.tld", referrer: "http://sub.domain.tld/

The nginx inside the container does work and answers to http requests (I jexec'd into the jail and tried)

In Tredlyfile I have:

tcpInPort=80
tcpInPort=443
tcpOutPort=80
tcpOutPort=443
udpOutPort=53
udpOutPort=21
dinapappor commented 8 years ago

Log

File transferred - /tredly/ptn/prod/tmp/mattermost/LICENSE.txt
File transferred - /tredly/ptn/prod/tmp/mattermost/README.md
File transferred - /tredly/ptn/prod/tmp/mattermost/Tredlyfile
File transferred - /tredly/ptn/prod/tmp/mattermost/config.py
File transferred - /tredly/ptn/prod/tmp/mattermost/nginx.conf
File transferred - /tredly/ptn/prod/tmp/mattermost/payload.py
File transferred - /tredly/ptn/prod/tmp/mattermost/preview.png
File transferred - /tredly/ptn/prod/tmp/mattermost/server.py
File transferred - /tredly/ptn/prod/tmp/mattermost/star.domain.tld/server.crt
File transferred - /tredly/ptn/prod/tmp/mattermost/star.domain.tld/server.key
============================================================
Replacing Container mattermost-github with mattermost-github

======================================
Creating Container - mattermost-github

Creation started at 13/05/2016 14:26:23 +0000
mattermost-github allocated IP 10.99.133.126/16
mattermost-github has DNS set to IP(s) 10.99.255.254
Setting resource limits
➜ maxCpu property value was not set. Defaulting to unlimited.
➜ maxRam property value was not set. Defaulting to unlimited.
➜ maxHdd property value was not set. Defaulting to unlimited.
Configuring firewall for mattermost-github
✔ Success
Updating package database
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
✔ Success
Updating container's pkg catalogue...
✔ Success
Installing: www/py-flask and its dependencies
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking integrity... done (0 conflicting)
The following 12 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
    py27-Flask: 0.10.1
    py27-itsdangerous: 0.24
    python27: 2.7.11_2
    libffi: 3.2.1
    indexinfo: 0.2.4
    gettext-runtime: 0.19.7
    py27-setuptools27: 20.0
    py27-werkzeug: 0.11.3
    py27-Jinja2: 2.8
    py27-Babel: 2.2.0_1
    py27-pytz: 2016.1,1
    py27-MarkupSafe: 0.23

The process will require 94 MiB more space.
[mattermost-github] [1/12] Installing indexinfo-0.2.4...
[mattermost-github] [1/12] Extracting indexinfo-0.2.4: .... done
[mattermost-github] [2/12] Installing libffi-3.2.1...
[mattermost-github] [2/12] Extracting libffi-3.2.1: .......... done
[mattermost-github] [3/12] Installing gettext-runtime-0.19.7...
[mattermost-github] [3/12] Extracting gettext-runtime-0.19.7: .......... done
[mattermost-github] [4/12] Installing python27-2.7.11_2...
[mattermost-github] [4/12] Extracting python27-2.7.11_2: .......... done
[mattermost-github] [5/12] Installing py27-setuptools27-20.0...
[mattermost-github] [5/12] Extracting py27-setuptools27-20.0: .......... done
[mattermost-github] [6/12] Installing py27-pytz-2016.1,1...
[mattermost-github] [6/12] Extracting py27-pytz-2016.1,1: .......... done
[mattermost-github] [7/12] Installing py27-Babel-2.2.0_1...
[mattermost-github] [7/12] Extracting py27-Babel-2.2.0_1: .......... done
[mattermost-github] [8/12] Installing py27-MarkupSafe-0.23...
[mattermost-github] [8/12] Extracting py27-MarkupSafe-0.23: .......... done
[mattermost-github] [9/12] Installing py27-itsdangerous-0.24...
[mattermost-github] [9/12] Extracting py27-itsdangerous-0.24: .......... done
[mattermost-github] [10/12] Installing py27-werkzeug-0.11.3...
[mattermost-github] [10/12] Extracting py27-werkzeug-0.11.3: .......... done
[mattermost-github] [11/12] Installing py27-Jinja2-2.8...
[mattermost-github] [11/12] Extracting py27-Jinja2-2.8: .......... done
[mattermost-github] [12/12] Installing py27-Flask-0.10.1...
[mattermost-github] [12/12] Extracting py27-Flask-0.10.1: .......... done
Message from python27-2.7.11_2:
===========================================================================

Note that some standard Python modules are provided as separate ports
as they require additional dependencies. They are available as:

bsddb           databases/py-bsddb
gdbm            databases/py-gdbm
sqlite3         databases/py-sqlite3
tkinter         x11-toolkits/py-tkinter

===========================================================================
✔ Success
Installing: www/py-requests and its dependencies
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
    py27-requests: 2.9.1

The process will require 3 MiB more space.
[mattermost-github] [1/1] Installing py27-requests-2.9.1...
[mattermost-github] [1/1] Extracting py27-requests-2.9.1: .......... done
✔ Success
Installing: www/nginx and its dependencies
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking integrity... done (0 conflicting)
The following 2 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
    nginx: 1.8.1,2
    pcre: 8.38_1

The process will require 6 MiB more space.
[mattermost-github] [1/2] Installing pcre-8.38_1...
[mattermost-github] [1/2] Extracting pcre-8.38_1: .......... done
[mattermost-github] [2/2] Installing nginx-1.8.1,2...
===> Creating users and/or groups.
Using existing group 'www'.
Using existing user 'www'.
[mattermost-github] [2/2] Extracting nginx-1.8.1,2: .......... done
✔ Success
Copying Container Data "/config.py" to "/tmp/"
✔ Success
Copying Container Data "/payload.py" to "/tmp/"
✔ Success
Copying Container Data "/server.py" to "/tmp/"
✔ Success
Copying Container Data "/nginx.conf" to "/usr/local/etc/nginx/"
✔ Success
Copying Partition Data "/star.domain.tld" to "/usr/local/etc/nginx/ssl"
✔ Success
Running onStart command: " python2.7 /tmp/server.py &"
✔ Success
Running onStart command: "echo 'nginx_enable="YES"'>>/etc/rc.conf "
✔ Success
Running onStart command: "service nginx start"
Performing sanity check on nginx configuration:
Starting nginx.
✔ Success
Creating onStop script
Adding container to DNS
✔ Success
Configuring layer 7 Proxy (HTTP) for mattermost-github
✔ Success
Reloading DNS server
✔ Success
✔ Creation completed at 13/05/2016 14:26:33 +0000
✔ Total time taken: 10 seconds
============================================
Destroying Container - mattermost-github-OLD

Destruction started at 13/05/2016 14:26:33 +0000
mattermost-github-OLD has IP address 10.99.209.52/16
Removing container from DNS
✔ Success
Removing url registration from DNS
✔ Success
Removing container networking
✔ Success
Stopping container mattermost-github-OLD
✔ Success
Destroying container mattermost-github-OLD
✔ Success
Updating container group firewall rules
✔ Success
Reloading DNS server
✔ Success
Reloading Layer 7 Proxy
✔ Success
✔ Destruction completed at 13/05/2016 14:26:44 +0000
✔ Container uptime: 17 minutes 33 seconds 
✔ Total time taken: 11 seconds
dinapappor commented 8 years ago

ipfw list:

ipfw list
00005 allow ip from any to any via lo0
00008 allow log logamount 5 ip from table(10) to any via table(11)
00009 deny log logamount 5 ip from table(1) to table(10) via table(2)
00100 nat 1 log logamount 5 ip from any to table(5) recv table(6)
00101 check-state log logamount 5
00110 allow log logamount 5 icmp from any to table(5) in via table(6) keep-state
00111 allow log logamount 5 tcp from any to table(5) dst-port 65222 in via table(6) setup keep-state
00112 allow log logamount 5 icmp from table(5) to any out via table(6) keep-state
00113 allow log logamount 5 tcp from table(5) to any dst-port 53,80,443,22,65222 out via table(6) setup keep-state
00114 allow log logamount 5 udp from table(5) to any dst-port 53,123 out via table(6) keep-state
00150 allow log logamount 5 tcp from table(20) to table(5) dst-port 65223 in via table(6) setup keep-state
00200 skipto 65510 log logamount 5 tcp from any to table(10) recv table(6) setup keep-state
00201 skipto 65510 log logamount 5 udp from any to table(10) recv table(6) keep-state
00202 skipto 65510 log logamount 5 tcp from table(10) to not table(10) xmit table(6) setup keep-state
00203 skipto 65510 log logamount 5 udp from table(10) to not table(10) xmit table(6) keep-state
00209 allow log logamount 5 ip from any to any via table(2) keep-state
00211 allow log logamount 5 ip from any to table(1) in via table(6) setup keep-state
65501 deny log logamount 5 ip from any to any
65510 nat 1 log logamount 5 ip from table(10) to any xmit table(6) keep-state
65511 allow log logamount 5 ip from table(5) to any xmit table(6) keep-state
65512 allow log logamount 5 ip from any to table(10) recv table(6) keep-state
65513 deny log logamount 5 ip from any to any
65535 deny ip from any to any
dinapappor commented 8 years ago

Alright, think I found the problem. Seems inside the container the rules are not added

/usr/local/etc/ipfw.rules only contains /usr/bin/env sh

Disabling ipfw inside the container with service ipfw stop let's the traffic through.

nathanaherne commented 8 years ago

If you turn on IPFW in the container and run IPFW list, what is the output?

laurieodgers commented 8 years ago

Looks like a bug with containergroup firewall rules - if you remove containergroup= from the Tredlyfile, do the firewall rules in the container get added to /usr/local/etc/ipfw.rules?

dinapappor commented 8 years ago

Yes, that works. Tried removing it, traffic passed through like it should.

Adding the containergroup again and once again there was no passing of traffic to the container.

Output of ipfw list and cat'ing the ipfw.rules:

root@mattermost-github:/ # cat /usr/local/etc/ipfw.rules 
#!/usr/bin/env sh

root@mattermost-github:/ # ipfw list
00100 allow log tcp from table(2) to 10.99.109.233 dst-port 80,443 in via vnet0 setup keep-state
00200 allow log tcp from table(3) to 10.99.109.233 dst-port 80,443 in via vnet0 setup keep-state
00300 allow log tcp from any to 10.99.109.233 dst-port 80,443 in via vnet0 setup keep-state
00400 allow log tcp from 10.99.109.233 to any dst-port 80 out via vnet0 setup keep-state
00500 allow log tcp from 10.99.109.233 to any dst-port 443 out via vnet0 setup keep-state
00600 allow log udp from 10.99.109.233 to any dst-port 53 out via vnet0 keep-state
00700 allow log tcp from 10.99.109.233 to any dst-port 80,443 out via vnet0 setup keep-state
00800 allow log udp from 10.99.109.233 to any dst-port 53,21 out via vnet0 keep-state
00900 allow log tcp from 10.99.255.254 to 10.99.109.233 dst-port 443 in via vnet0 setup keep-state
01000 allow log udp from 10.99.109.233 to 10.99.255.254 dst-port 53 out via vnet0 keep-state
01100 allow log udp from 10.99.109.233 to 10.99.109.233 via vnet0 keep-state
01200 allow log tcp from 10.99.109.233 to 10.99.109.233 via vnet0 setup keep-state
01300 allow log ip from any to any via lo0
65535 deny ip from any to any

While the suggested workaround worked with http it does still not work with https. :(

laurieodgers commented 8 years ago

The following rule suggests that HTTPS is allowed to your container from the proxy: 00300 allow log tcp from any to 10.99.109.233 dst-port 80,443 in via vnet0 setup keep-state

Do you have HTTPS enabled within your container? I'm not 100% on how the layer 7 proxy will react to a self signed certificate because its untested so far; but if its a problem I am sure we can sort something out to deal with it.

dinapappor commented 8 years ago

These are genuine rapidssl wildcard certs. Both in reverse proxy and in the container. Den 14 maj 2016 15:55 skrev "Laurie Odgers" notifications@github.com:

The following rule suggests that HTTPS is allowed to your container from the proxy: 00300 allow log tcp from any to 10.99.109.233 dst-port 80,443 in via vnet0 setup keep-state

Do you have HTTPS enabled within your container? I'm not 100% on how the layer 7 proxy will react to a self signed certificate because its untested so far; but if its a problem I am sure we can sort something out to deal with it.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/tredly/tredly-host/issues/36#issuecomment-219221634

laurieodgers commented 8 years ago

I have found an issue with the recreation of the containergroup ipfw table within each container after a container is destroyed (array not declared local).

Other than this unfortunately I cannot reproduce the problem. If this patch 0.10.4 doesn't fix the issue, can you please paste in your tredlyfile, plus any steps to reproduce the problem (create/destroy/replace etc), and these files from the host:

And the following tables from the container:

dinapappor commented 8 years ago

Hi, updated tredly-build. HTTPs still does not work. I have completely removed the container - it was the only one running on the host.

I then pushed the container.

#############################
#### Tredlyfile version number
versionNumber=0.10.0
#############################

# You can use variables within this file which should be interpreted by your
# Code repository, e.g. gitlabs or Bamboo. You can also configure Tredly-Parse
# to interpret values or override values.

################################################################################
#### CONTAINER NAME AND CONTAINER GROUP

## Container Name [required]
# Unique per Partition
# Allowed values: text, integer, underscore (_) and dash (-)
# This should contain your AppName and version number. The hostname of this
# container will be equal to this value.
containerName=mattermost-github

## Container Group [optional]
# Allowed values: text, integer, underscore (_) and dash (-)
# Only use this option if this container is part of a group of containers

## Container Options within Container Group [optional]
# ** Note: required if containerGroup is specified
# Allowed values: a number from 1 (most important) to 99 (least important)
# Default: 1
# Order in which this container is started within your containerGroup
# Please note that if you update one container within your containerGroup, all
# containers will be rebuilt
startOrder=1

## Replicate Container [optional]
# When set to yes, this allows a container to be replicated within its
# containerGroup
# ** Note: required if containerGroup is specified
# Allowed values: yes, no
# Default: no
replicate=no

################################################################################
#### BUILD OPTIONS

## Publish container automatically [required]
# When set to yes, on push to Tredly-Host this container will be automatically
# validated and built. The updated container will replace the current container.
# To automatically build integration branch but manually build prod branch, you
# can use a variable.
# Allowed values: yes, no
# Default: yes
publish=yes

################################################################################
#### TECHNICAL OPTIONS

## Container options [optional]
# This options will be set when the container is created. Supports multiple
# options; one per line.
# Allowed values: allow_sysvipc=1, allow_raw_sockets=true
# Default: none
# To allow ping/traceroute set allow_raw_sockets=true.
# PostgreSQL requires allow_sysvipc=1.
# Note: Both of these options may have security implications.
TechnicalOptions=

################################################################################
### RESOURCE LIMITS
## Limit resources used by this container.
# It is recommended that resource limits so that a misconfigured container, or
# buggy software, does not effect other containers on the same host.

## Container Maximum CPU [optional]
# Specify the total number of cores/percent this container can utilize
# Allowed values: integer + %
# Default: unlimited
maxCpu=50%

## Container Hard Disk Drive Size [optional]
# Specify the total amount of disk space, in Gigabytes (GB) this container
# will be allocated
# Allowed values: integer
# Default: unlimited
maxHdd=0,5

## Container Maximum RAM [optional]
# Specify the total amount of RAM, in Gigabytes (GB) this container can utilize
# Allowed values: integer
# Default: unlimited
maxRam=0,256

################################################################################
#### FIREWALL AND PROXY CONFIGURATION

## HTTP/HTTPS (layer7) Proxy Configuration [optional]
# Group of properties for URL(s) the container services and responds to.
# Each group has 5 different properties that can be set. Each property in the
# set must be proceeded by a number (see examples below):
#
# 1. url
#     This is a full URL, including any additional path, but excluding the
#     protocol. Any traffic that is received by the host for this particular URL
#     will be directed to this container. Only 1 URL can be specified per group
#     of properties.
#     e.g. url1=www.example.com/blah
#
# 2. urlRedirect [optional] [will be implemented for 0.10.0]
#     Redirect any requests to the main url specified for this group. Allows
#     multiple values; place each one a new line. You must include the protocol.
#     e.g. urlRedirect1=http://example.com
#          urlRedirect1=https://otherexample.com
#
# 3. urlRedirectCert [optional]
#     SSL certificate to apply to a http redirect "from" URL. Due to the HTTPS
#     handshake occurring before the redirect message is sent to the browser,
#     you must specify a certificate for the redirect "from" URL. For example
#     if you specify url1Redirect=https://www2.example.com then you must specify
#     a certificate for this URL to use.
#     e.g. urlRedirectCert=star.example.com
#
# 4. urlCert [optional]
#     SSL configuration for URL - corresponds to a ssl definition file within
#     nginx/sslconfig. urlCert is folder name containing server.crt and
#     server.key for this URL. If URL has urlCert entered proxy will
#     automatically redirect HTTP traffic to HTTPS URL. If urlCert is not
#     specified connections to HTTPS will result in 404 error.
#     e.g. urlCert1=star.example.com
#
# 5. urlWebsocket [optional]
#     Enable Websockets for this URL. Additional headers will be set by the
#     layer7 proxy to allow Websocket connections to be persistent, and
#     connection upgrades of http:// to ws:// or https:// to wss:// will occur
#     automatically.
#     Allowed values: yes, no
#     Default: no
#     e.g. urlWebsocket1=yes
#
# 6. urlMaxFileSize [optional]
#     Set the maximum allowable upload size, in megabytes (m) or gigabytes (g)
#     for this URL.
#     Allowed values: value between 1m and 2g
#     Default: 1m
#     e.g. urlMaxFileSize1=10m
#
url1=github-int.domain.tld
url1Cert=partition/star.domain.tld/

## IPv4 Proxy (layer4) [optional]
# Layer4 proxy allows the container to use the Hosts external IP and forward
# traffic to your container on specific ports. Make sure the ports selected in
# tcpInPort and udpInPort options are available on the host.
# ** Note: Cannot be used on port 80/443 (HTTP/HTTPS) as internal Proxy handles
# this traffic. If layer4Proxy is set to yes, tcpInPort and udpInPort options
# will be forwarded directly to your container from the hosts external IP.
# Allowed values: yes, no
# Default: no
layer4Proxy=

## Firewall - Incoming & Outgoing Ports
# Required if you want this container to be able to communicate with other
# containers or the outside world.
# Allowed values: integer, any (all ports) or blank
# One port per line
# ** Note: tcpOutPort=80, tcpOutPort=443 and udpOutPort=53 are required values.
# ** Note: if your container services a URL make sure tcpInPort=80 or
# tcpInPort=443 is set so this containers firewall is open on this port and can
## receive traffic from the internal HTTP proxy.
tcpInPort=80
tcpInPort=443
tcpOutPort=80
tcpOutPort=443
udpOutPort=53

## IPv4 Whitelist [optional]
# Restrict traffic to this container to an ip address or network range.
# Allowed values: valid IPv4 address and subnet
# ** Note: Use containerGroup To allow communications between containers
# in a partition
# Default: any
#ipv4Whitelist=

################################################################################
#### CUSTOM DNS SERVERS [optional]
# Specify alternative DNS servers for this container to use. If left blank, or
# not specified, then the internal DNS server will be used (recommended).
# Allowed values: valid IP address or none
customDNS=

################################################################################
### Custom Commands and Operations
## Each command or operation in the section is run in order. All are
# optional and each must be placed on a new line. The available options are:
#
# 1. onStart
#       This command will be run when the container is started.
#       e.g. onStart= mkdir /usr/local/pgsql
#
# 2. onStop:
#       This command is run when the container is stopped, and before it
#       is destroyed. onStop commands should always be listed last.
#       e.g. onStop= service postgresql stop
#
# 3. installPackage
#       When this container is built, the package listed will be installed.
#       Any dependencies of the package specified will be installed.
#       e.g. installPackage=nginx
#
# 4. fileFolderMapping
#       Consists of two parts: the source and destination.
#       Source is relative to the container root (/) or
#       partition root (partition/) on the host. Destination
#       is the absolute path within the container where the file
#       or folder will be copied. Folders will be created in
#       the container first if they do not exist
#       e.g. fileFolderMapping= postgresql.conf /usr/local/pgsql/data/
#            fileFolderMapping=partition/mySSLCerts /usr/local/etc/nginx/ssl
#
# ***** Below is a example of a typical Nginx and PostgreSQL install *****

installPackage=www/py-flask
installPackage=www/py-requests
installPackage=www/nginx

fileFolderMapping=/config.py /tmp/
fileFolderMapping=/payload.py /tmp/
fileFolderMapping=/server.py /tmp/
fileFolderMapping=/nginx.conf /usr/local/etc/nginx/
fileFolderMapping=partition/star.domain.tld /usr/local/etc/nginx/ssl

# Start the mattermost integration
onStart= python2.7 /tmp/server.py &
onStart=echo 'nginx_enable="YES"'>>/etc/rc.conf 
onStart=service nginx start

# Commands run when container stops or is terminated (one per line, optional)
onStop=

# ***** End Example Section *****

The push of the container executes without errors.

The contents of nginx/server_name/

root@tredly01:/tmp/tredly-build # cat /usr/local/etc/nginx/server_name/http-github-int_domain_tld 
server {
    server_name github-int.domain.tld;
    listen 10.99.255.254:80;
    location / {
        include proxy_pass/http_https;
        proxy_bind 10.99.255.254;
        proxy_pass http://http-github-int_domain_tld;
    }
    location /tredly_error_docs {
        alias /usr/local/etc/nginx/tredly_error_docs;
        log_not_found off;
        access_log off;
    }
    error_page 404 /tredly_error_docs/404.html;
}

As you can see it does not listen on https. :I

Upstream:

upstream http-github-int_domain_tld {
    server 10.99.196.95:80;
}

While your jexec ipfw commands did not work I tried them inside the container of which none produced output. It's the same if I run them outside the table. The /usr/local/etc/ipfw.rules is still empty except for#!/usr/bin/env sh`.

The flask app is a simple app listening on port 8080. And nginx is also forwarding to it (it works when I jexec into the jail and curl/fetch).

nathanaherne commented 8 years ago

I am surprised you are having issues with this, we create hundreds of HTTPS containers a day without issue. There has got to be a simple explanation for this issue.

At first look:

dinapappor commented 8 years ago

512 MiB of hdd and 256 MiB of ram

fileFolderMapping is for the container to be able to have it copied to inside the container. url1Cert is located inside /usr/local/etc/nginx/ssl/partitionname. Atleast that's where it seems to be looking.

nathanaherne commented 8 years ago

I think its these values: maxHdd=0,5 maxRam=0,256

They are directly before the URL setup and the commas are possibly creating the issue.

You have found a bug though, I have no idea how you would set 500MB of HDD and 256MB RAM as we have only ever used full GB. I will lodge a feature request for this.

dinapappor commented 8 years ago

Since this box is empty, sans this one container I could provide access to it for easier debugging if that is of interest, it's on AWS.

(Had to build my own ami with zfs on root, and then build tredly on it. Fun exercise! No for real, it was fun!)

nathanaherne commented 8 years ago

Can you recreate the container with maxHdd=1 maxRam=1 This may solve your problem in the short term. I have added a feature request to have less than 1GB RAM and 1GB HDD.

dinapappor commented 8 years ago

I removed the entries all together (prefixed them with a #). It didnt solve my problem though. :/

nathanaherne commented 8 years ago

Ok, well that isn't good. I will have to have a think about it and if I cannot come up with something, in about 11 hours the Tredly developers will be back at work and will work out why this issue is happening.

laurieodgers commented 8 years ago

Thanks for the Tredlyfile dinapappor - this made troubleshooting much easier.

It looks like the trailing slash in url1Cert= was the culprit - I have applied a fix to strip the trailing slash and it will be merged in shortly.

dinapappor commented 8 years ago

Hi, sorry for replying to this yet again.

The issue seems not to have been any ssl at all. It had to do with tcp segmentation offloading (TSO).

This is a known problem with FreeBSD on AWS/EC2, I solved my problem with net.inet.tcp.tso=0 sorry for providing you guys with a (possibly?) false bug report. That sysctl should probably be added to the default install since IPFW is much like PF totally incompatible with TSO being on.

Sorry. :(

nathanaherne commented 8 years ago

Thanks for coming back and telling us. We already disable TSO but was only doing it for our particular use case - which is why we hadn't seen your issue. We will update the TSO disable to do it for all cards.

laurieodgers commented 8 years ago

TSO is now disabled system wide - if you need this please pull from the integration branch (change is not currently in master until next release).

With respect to the SSL bug, it was actually a bug in tredly-build which I was able to reproduce so thanks for the report :)