Steam download becomes slow and/or dies after some time.

[northys]

Checklist:

• [x] I'm using the official portmaster release (i.e no custom builds)
• [x] I searched for similar/existing issues first.
• [x] My issue is not mentioned in the Known Issues section of my OS

What happened:

Today I reinstalled my Ubuntu 20.04 to dualboot combo Windows + Fedora (more on that later, systemd unit for fedora / general linux has to be updated, portmaster run manually works great!) so I can play remastered Mafia. Even before reinstall I noticedthe network is really slow. I downloaded the 5G Windows ISO for over 1.5 hour. I was surprised and confused at the same time when networking went fine and at full speed testing from fresh windows installation. I downloaded steam, firefox, logged in, started downloading Mafia. Everything was a lot smoother and faster from what I can say. Game downloader in steam says 12M/s peak. My ISP plan is 100Mbps. Great! Installed portaster on Windows and rebooted. Steam didn't go over 6M/s. But what was more interesting - after few minutes it crashed to 0 and recovered after a half minute or so. Then it crashed again. Faster than the first time.

What you expected to happen:

Nothing =)

How to reproduce it (as minimally and precisely as possible):

W10 + portmaster + steam game download @ good connection

Anything else we need to know?:

Environment:

tested on windows but i think linux is also affected

Portmaster Version:

0.5.6 - currently available latest, installed today

Operating System:

• [ ] Windows 7
• [ ] Windows 8/8.1
• [x] Windows 10
• [x] Linux

If applicable you can provide related sections from the log files and ensure to remove sensitive or otherwise private information.

latest log by timestamp - https://bin.privacytools.io/?ccf844083118d6b8#9uY7nJAcmUUxwW2sPmvU2Y/Uyo1z4LByO4+Cs2r0WXo= error log file is 0b.

[dhaavi]

Thanks for the report!

For clarification: Did you experience Portmaster slowing down the network on both Linux and Windows? Or was it Windows only?

This, in general, sounds a lot like you had the setting "Permanent Verdicts" disabled. This option should always be enabled in normal circumstances, as it "gives the connection back" to the Kernel after it has decided what to do with it. From there, the Portmaster does not touch the connection anymore and you should be experienced the exact same speed as without the Portmaster.

If that is not the case, then this is going to be a lot more interesting. Further things I can think of:

• There is a problem with handling network connections at the firewall integration level. Nothing has changed here in the last months, so this seems rather unlikely.
• With the Portmaster's DNS resolving (ie. Cloudflare?), Steam may select a different server to download from that is a lot slower.

Could you please diversify your speed testing to ensure that this is not only affecting one service? Eg. try multiple speed test sites, or even better, transfer a big file to a fast network storage in your LAN to test the maximum network speed of the device directly.

[dhaavi]

@ppacher and I have looked further into this, but we could not identify anything that would cause this behavior, except for the "Permanent Verdicts" setting.

I also could not find anything unusual in the logs. This is quite confusing. It would be great if you could do more testing in that regard and give us more information about this.

[northys]

My laptop configuration and network info so I don't have to mention it multiple times in my answers: Intel Core i7 @3.8GHz Turbo boost. I think it's the 8th generation, 2 years old arch. 16G RAM NVME 100Mbps fiber to the home, laptop is connected using cheap xiaomi wifi router

So let's say computing power and local networking is not the issue for now...

For clarification: Did you experience Portmaster slowing down the network on both Linux and Windows? Or was it Windows only?

Both. The networking was ridicuously slow on my Ubuntu before reinstall. It took me 1.5h to download the 5G Windows ISO from Microsoft servers. Fedora ISO was much faster but it didn't go over 3MB/s.

I couldn't even believe my parents are not angry because we all were browsing the internet and my father is often angry when the internet is slow for some reason (e.g. my home server is doing backups...). But the answer was that the internet it just fine. When I downloaded the fedora on Windows (no portmaster) it went full speed - 11MB/s or so.

I can't confirm this on my ubuntu because it's already wiped. But I have backup of /var/lib/pormaster few minutes before wipe.

This, in general, sounds a lot like you had the setting "Permanent Verdicts" disabled. This option should always be enabled in normal circumstances, as it "gives the connection back" to the Kernel after it has decided what to do with it. From there, the Portmaster does not touch the connection anymore and you should be experienced the exact same speed as without the Portmaster.

I can't tell you because the dashboard is not working right now on Windows. It used to work yesterday though. If I can send you some kind of config from Ubuntu please tell me the location. I do have backup of my Ubuntu installation and I think you store it somewhere as json.

There is a problem with handling network connections at the firewall integration level. Nothing has changed here in the last months, so this seems rather unlikely.

It can be slowing down our network connections all the sime since you've first run the dev version of portmaster on your laptop. Nobody just noticed yet?

BTW I think this is the problem with networking I reported from our office and from hotel. Both of those WiFis are limited to around (omg I need to learn to write a tilde sign on WIndows) 8Mbps so when you divide it by 2 the internet becomes barelly usable... And this is the feeling I do have for a long time. I just didn't try to turn off portmaster. Maybe it is the same think I reported in DNS not working at all from train (#132 - https://github.com/safing/portmaster/issues/132#issuecomment-696652439)? When it stopped suddenly working while using USB tethering with LTE?

With the Portmaster's DNS resolving (ie. Cloudflare?), Steam may select a different server to download from that is a lot slower.

That would make it slow but still working. Currently it drops to 0MB/s for a minute or so and then it starts downloading again.

EDIT: IIRC my router has set 1.1.1.1 as primary and 8.8.8.8 as backup so it would work bad even before installing portmaster. Cloudflare has edge in Prague so I don't expect it resolved Steam CDN edges from Singapore.

Could you please diversify your speed testing to ensure that this is not only affecting one service? Eg. try multiple speed test sites, or even better, transfer a big file to a fast network storage in your LAN to test the maximum network speed of the device directly.

Actually I can't. Steam is very very very specific service when it comes to downloading speed. It is the only service where my friends (let's say 8 years ago) always told me it is going full speed. Wether they do have shitty local Wi-Fi provider, fiber to the home or anything else. Back to my childhood when I had 8Mbps Wi-Fi connection to the internet the steam was the only one service which downloaded 1MB/s. Even speedtest.net measured just 500kB/s (4Mbps) and YouTUBE (I expect their CDN is at least as good as the steam one) couldn't run 1080p video smoothly.

I don't know what Steam does and why they have so fast CDN edges but I think they utilize a lot of concurent connections each throtled @ let's say 500kB/s and it runs as much parallel connections as it can handle. This is nothing I can confirm using any available speed testers. I can just confirm that it acts as bad as it did yesterday.

P.S. please never read my posts from e-mail. I always edit my post like 20-30 times before you read it for the first time at work in the morning 😆

OT: If you are going to test it you can test it using Mafia: definitive edition. It's great remastered Czech classic which was originally made by few people in Brno =)

It's so good that I've spent almost 11 hours of creating windows bootable stick (woeusb not working on Ubuntu 20.04, default startup disk creator is for Ubuntu only, dd didn't work...) so I can play game of my childhood again.

[dhaavi]

Ok. Having the config files would be a good start. Could you upload both the config file from the Ubuntu you had and the one from Windows? They are at the root of the data directory. Just click the "Open Data Directory" in the UI, if available.

I know that speedtests are always to be taken with a grain of salt, but having another source of data is always helpful. You could also check out the new one from Cloudflare, they give some nice insights: https://speed.cloudflare.com/

[northys]

I don't see the config file?

I'll just left the original paste from windows console here so everybody can see how stupid is this OS:

C:\ProgramData\Safing\Portmaster>tree /f                                                                                                                                                                                                     Folder PATH listing                                                                                                                                                                                                                          Volume serial number is 221A-43E3                                                                                                                                                                                                            C:.                                                                                                                                                                                                                                          │   portmaster-start.exe                                                                                                                                                                                                                     │   portmaster-uninstaller.exe                                                                                                                                                                                                               │   portmaster.ico                                                                                                                                                                                                                           │                                                                                                                                                                                                                                            ├───databases                                                                                                                                                                                                                                │   ├───cache                                                                                                                                                                                                                                │   │   └───bbolt                                                                                                                                                                                                                            │   │           db.bbolt                                                                                                                                                                                                                     │   │                                                                                                                                                                                                                                        │   └───core                                                                                                                                                                                                                                 │       └───bbolt                                                                                                                                                                                                                            │               db.bbolt                                                                                                                                                                                                                     │                                                                                                                                                                                                                                            ├───exec                                                                                                                                                                                                                                     ├───logs                                                                                                                                                                                                                                     │   │   2020-09-28-23-30-50.log                                                                                                                                                                                                              │   │   2020-09-28-23-33-03.error.log                                                                                                                                                                                                        │   │   2020-09-28-23-33-03.log                                                                                                                                                                                                              │   │   2020-09-29-00-07-10.log                                                                                                                                                                                                              │   │   2020-09-29-00-26-33.log                                                                                                                                                                                                              │   │   2020-09-29-00-59-20.log                                                                                                                                                                                                              │   │   2020-09-29-20-20-09.log                                                                                                                                                                                                              │   │   2020-09-30-18-00-13.log                                                                                                                                                                                                              │   │   2020-09-30-18-09-06.log                                                                                                                                                                                                              │   │                                                                                                                                                                                                                                        │   ├───app                                                                                                                                                                                                                                  │   │       2020-09-29-20-19-01.error.log                                                                                                                                                                                                    │   │                                                                                                                                                                                                                                        │   ├───notifier                                                                                                                                                                                                                             │   │       2020-09-28-23-32-04.error.log                                                                                                                                                                                                    │   │       2020-09-28-23-33-21.error.log                                                                                                                                                                                                    │   │       2020-09-29-00-07-28.error.log                                                                                                                                                                                                    │   │       2020-09-29-00-58-23.error.log                                                                                                                                                                                                    │   │       2020-09-29-00-59-46.error.log                                                                                                                                                                                                    │   │       2020-09-29-20-20-22.error.log                                                                                                                                                                                                    │   │       2020-09-30-18-00-47.error.log                                                                                                                                                                                                    │   │                                                                                                                                                                                                                                        │   └───start                                                                                                                                                                                                                                │           2020-09-28-23-30-50.log                                                                                                                                                                                                          │           2020-09-28-23-31-11.log                                                                                                                                                                                                          │           2020-09-28-23-32-04.log                                                                                                                                                                                                          │           2020-09-28-23-33-03.log                                                                                                                                                                                                          │           2020-09-28-23-33-21.log                                                                                                                                                                                                          │           2020-09-29-00-07-10.log                                                                                                                                                                                                          │           2020-09-29-00-07-28.log                                                                                                                                                                                                          │           2020-09-29-00-07-58.log                                                                                                                                                                                                          │           2020-09-29-00-26-33.log                                                                                                                                                                                                          │           2020-09-29-00-26-49.log                                                                                                                                                                                                          │           2020-09-29-00-28-45.log                                                                                                                                                                                                          │           2020-09-29-00-59-20.log                                                                                                                                                                                                          │           2020-09-29-00-59-46.log                                                                                                                                                                                                          │           2020-09-29-01-26-10.log                                                                                                                                                                                                          │           2020-09-29-01-26-34.log                                                                                                                                                                                                          │           2020-09-29-20-19-01.log                                                                                                                                                                                                          │           2020-09-29-20-20-09.log                                                                                                                                                                                                          │           2020-09-29-20-20-22.log                                                                                                                                                                                                          │           2020-09-29-20-20-42.log                                                                                                                                                                                                          │           2020-09-30-18-00-13.log                                                                                                                                                                                                          │           2020-09-30-18-00-47.log                                                                                                                                                                                                          │           2020-09-30-18-09-06.log                                                                                                                                                                                                          │           2020-09-30-18-09-09.log                                                                                                                                                                                                          │                                                                                                                                                                                                                                            └───updates                                                                                                                                                                                                                                      │   beta.json                                                                                                                                                                                                                                │   stable.json                                                                                                                                                                                                                              │                                                                                                                                                                                                                                            ├───all                                                                                                                                                                                                                                      │   ├───intel                                                                                                                                                                                                                                │   │   │   intel.json                                                                                                                                                                                                                       │   │   │                                                                                                                                                                                                                                    │   │   ├───geoip                                                                                                                                                                                                                            │   │   │       geoipv4_v20200514-6-49.mmdb                                                                                                                                                                                                  │   │   │       geoipv4_v20200514-6-49.mmdb.gz                                                                                                                                                                                               │   │   │       geoipv6_v20200514-6-49.mmdb                                                                                                                                                                                                  │   │   │       geoipv6_v20200514-6-49.mmdb.gz                                                                                                                                                                                               │   │   │                                                                                                                                                                                                                                    │   │   └───lists                                                                                                                                                                                                                            │   │           base_v20200923-7-58.dsdl                                                                                                                                                                                                     │   │           index_v2020-9-22.dsd                                                                                                                                                                                                         │   │           intermediate_v20200926-22-0.dsdl                                                                                                                                                                                             │   │           urgent_v20200928-23-0.dsdl                                                                                                                                                                                                   │   │           urgent_v20200929-0-0.dsdl                                                                                                                                                                                                    │   │           urgent_v20200929-20-0.dsdl                                                                                                                                                                                                   │   │           urgent_v20200930-17-0.dsdl                                                                                                                                                                                                   │   │           urgent_v20200930-18-0.dsdl                                                                                                                                                                                                   │   │                                                                                                                                                                                                                                        │   └───ui                                                                                                                                                                                                                                   │       └───modules                                                                                                                                                                                                                          │               assets_v0-1-5.zip                                                                                                                                                                                                            │               base_v0-2-10.zip                                                                                                                                                                                                             │                                                                                                                                                                                                                                            └───windows_amd64                                                                                                                                                                                                                                ├───app                                                                                                                                                                                                                                      │       portmaster-app_v0-1-10.exe                                                                                                                                                                                                           │                                                                                                                                                                                                                                            ├───core                                                                                                                                                                                                                                     │       portmaster-core_v0-5-6.exe                                                                                                                                                                                                           │                                                                                                                                                                                                                                            ├───kext                                                                                                                                                                                                                                     │       portmaster-kext_v1-0-3.dll                                                                                                                                                                                                           │       portmaster-kext_v1-0-3.sys                                                                                                                                                                                                           │                                                                                                                                                                                                                                            ├───notifier                                                                                                                                                                                                                                 │       portmaster-notifier_v0-1-9.exe                                                                                                                                                                                                       │       portmaster-snoretoast_v0-6-0.exe                                                                                                                                                                                                     │                                                                                                                                                                                                                                            └───start                                                                                                                                                                                                                                            portmaster-start_v0-4-2.exe     

Fixed line endings:

Folder PATH listing
Volume serial number is 221A-43E3
C:.
│   kunda.log
│   portmaster-start.exe
│   portmaster-uninstaller.exe
│   portmaster.ico
│   
├───databases
│   ├───cache
│   │   └───bbolt
│   │           db.bbolt
│   │           
│   └───core
│       └───bbolt
│               db.bbolt
│               
├───exec
├───logs
│   │   2020-09-28-23-30-50.log
│   │   2020-09-28-23-33-03.error.log
│   │   2020-09-28-23-33-03.log
│   │   2020-09-29-00-07-10.log
│   │   2020-09-29-00-26-33.log
│   │   2020-09-29-00-59-20.log
│   │   2020-09-29-20-20-09.log
│   │   2020-09-30-18-00-13.log
│   │   2020-09-30-18-09-06.log
│   │   
│   ├───app
│   │       2020-09-29-20-19-01.error.log
│   │       
│   ├───notifier
│   │       2020-09-28-23-32-04.error.log
│   │       2020-09-28-23-33-21.error.log
│   │       2020-09-29-00-07-28.error.log
│   │       2020-09-29-00-58-23.error.log
│   │       2020-09-29-00-59-46.error.log
│   │       2020-09-29-20-20-22.error.log
│   │       2020-09-30-18-00-47.error.log
│   │       
│   └───start
│           2020-09-28-23-30-50.log
│           2020-09-28-23-31-11.log
│           2020-09-28-23-32-04.log
│           2020-09-28-23-33-03.log
│           2020-09-28-23-33-21.log
│           2020-09-29-00-07-10.log
│           2020-09-29-00-07-28.log
│           2020-09-29-00-07-58.log
│           2020-09-29-00-26-33.log
│           2020-09-29-00-26-49.log
│           2020-09-29-00-28-45.log
│           2020-09-29-00-59-20.log
│           2020-09-29-00-59-46.log
│           2020-09-29-01-26-10.log
│           2020-09-29-01-26-34.log
│           2020-09-29-20-19-01.log
│           2020-09-29-20-20-09.log
│           2020-09-29-20-20-22.log
│           2020-09-29-20-20-42.log
│           2020-09-30-18-00-13.log
│           2020-09-30-18-00-47.log
│           2020-09-30-18-09-06.log
│           2020-09-30-18-09-09.log
│           
└───updates
    │   beta.json
    │   stable.json
    │   
    ├───all
    │   ├───intel
    │   │   │   intel.json
    │   │   │   
    │   │   ├───geoip
    │   │   │       geoipv4_v20200514-6-49.mmdb
    │   │   │       geoipv4_v20200514-6-49.mmdb.gz
    │   │   │       geoipv6_v20200514-6-49.mmdb
    │   │   │       geoipv6_v20200514-6-49.mmdb.gz
    │   │   │       
    │   │   └───lists
    │   │           base_v20200923-7-58.dsdl
    │   │           index_v2020-9-22.dsd
    │   │           intermediate_v20200926-22-0.dsdl
    │   │           urgent_v20200928-23-0.dsdl
    │   │           urgent_v20200929-0-0.dsdl
    │   │           urgent_v20200929-20-0.dsdl
    │   │           urgent_v20200930-17-0.dsdl
    │   │           urgent_v20200930-18-0.dsdl
    │   │           
    │   └───ui
    │       └───modules
    │               assets_v0-1-5.zip
    │               base_v0-2-10.zip
    │               
    └───windows_amd64
        ├───app
        │       portmaster-app_v0-1-10.exe
        │       
        ├───core
        │       portmaster-core_v0-5-6.exe
        │       
        ├───kext
        │       portmaster-kext_v1-0-3.dll
        │       portmaster-kext_v1-0-3.sys
        │       
        ├───notifier
        │       portmaster-notifier_v0-1-9.exe
        │       portmaster-snoretoast_v0-6-0.exe
        │       
        └───start
                portmaster-start_v0-4-2.exe

[dhaavi]

I don't see the config file?

If there is none, then you haven't changed any global settings.

Which now also does not give us any further clues... Hm. 🤔

[ErichDonGubler]

I just barely found this issue after a couple of frustrating weeks debugging my entire Internet connection at home! I installed PortMaster without taking time to configure it (because I'd forgotten to play with it, and that I'd installed it), and I'm having the same symptoms as OP on Windows 10 (version 1909, build 18363.1082 -- relatively recent). My connection, which is normally ~2 MB/s on Ethernet, goes down to ~200 KB/s when things ARE downloading...but often I'll get zero throughput for periods of minutes at a time, making it impossible to have use things like conferencing technologies. After finally deciding to try Safe Mode with Networking, I audited the software on my machine and did a prompt facepalm after realizing this was installed at about the time I started having issues.

So, as with OP's case indicated by @dhaavi, it seems that I also had no non-default configuration for PortMaster on my machine.

[northys]

If there is none, then you haven't changed any global settings.

@dhaavi makes sence. i told ya i can't access the dashboard :laughing:

Ubuntu 20.04 config from backup:

{
  "core": {
    "devMode": true,
    "expertiseLevel": "developer",
    "releaseLevel": "beta"
  },
  "dns": {
    "nameservers": [
      "dot://8.8.8.8",
      "dot://8.8.4.4",
      "dot://1.1.1.1"
    ]
  },
  "filter": {
    "endpoints": [
      "+ .cloudfront.net"
    ],
    "removeOutOfScopeDNS": 4
  },
  "spn": {
    "enable": false,
    "specialAccessCode": "********"
  }

@ErichDonGubler i'm happy there are more of us having the same issue. hope it makes the debugging process faster =)

[northys]

@dhaavi I can't confirm this behaviour using speed.cloudflare.com on Windows 10 running latest portmaster with default config.

5 measures with portmaster measured ~ 80-90/70-80Mbps 5 measures without portmaster measured ~ 90-95/80-90Mbps 5 another measures with portmaster measured ~ 85-95Mbps/75-85Mbps

so I don't think it affects downloads made in single thread

[dhaavi]

Thanks for chiming in here @ErichDonGubler! Sorry for the inconvenience this has caused. It's great to have a second person who has this problem. This will make it a lot easier to see what is going on here.

Ok, so there are multiple things we want to achieve right now:

---

1. Narrow down the things that might be causing this.

With two people, we can already start looking for overlapping environmental influences:

• @northys has reported the following things so far:
• both Ubuntu and Windows are affected
• zero throughput periods
• 100MBits Internet uplink
• Xiamoi WiFi router
• confirmed services affected:
  • Steam downloading (Windows only?)
  • Big single file download (ISO) via Browser (?) on both Linux and Windows - please confirm
• confirmed services not affected:
  • Speedtest (Cloudflare)
• default configuration (Windows)

@ErichDonGubler has mentioned so far:

• Window is affected
• zero throughput periods
• default configuration
• confirmed services affected: conferencing software mentioned, no hard facts yet

Please supply us with more information regarding this issue, especially if there are services that seem more affected than others.

---

2. Finding the scope when and where this problem applies.

We know about Steam and conferencing software. I'd like to specifically test different traffic types. It would be great if you could use iperf3 on the affected devices to get some network performance data.

Also, is the bandwidth impaired all the time? Or just at certain times or in certain places? How often do the "zero throughput periods" occur? Have you noticed any circumstances regarding that?

You can get iperf3 from here: https://software.es.net/iperf/obtaining.html Here are the commands to test TCP and UDP with both IPv4 and IPv6:

iperf3.exe -c 138.201.172.105 -p 56789
iperf3.exe -u -b 100M -c 138.201.172.105 -p 56789
iperf3.exe -c 2a01:4f8:c17:dddf::1 -p 56789
iperf3.exe -u -b 100M -c 2a01:4f8:c17:dddf::1 -p 56789

Please upload the (redacted) output of these commands to https://bin.privacytools.io/ and post a link here.

---

We are still trying out a couple things, but could not reproduce the issue yet. We did find something interesting however: If there are loads of connections, then the Portmaster becomes quite slow regarding responding to packets. But this behavior does not impact the bandwidth of new or already accepted connections in any way - they continue at full speed.

The delay created by the Portmaster could however lead to these zero throughput periods, if there are a lot, like thousands of new connections on the network.

[northys]

We are still trying out a couple things, but could not reproduce the issue yet. We did find something interesting however: If there are loads of connections, then the Portmaster becomes quite slow regarding responding to packets. But this behavior does not impact the bandwidth of new or already accepted connections in any way - they continue at full speed.

The delay created by the Portmaster could however lead to these zero throughput periods, if there are a lot, like thousands of new connections on the network.

I think it happens in Steam because it does a lot of concurent connections at the same time. IMHO it affected single connection download of win ISO in Ubuntu because I have a lot of stuff that connects to the internet all the time - steam, dropbox, nextcloud, telegram, slack, wireguard, monitoring, connectivity check... it is hard to find a quiet period in portmaster log.

Also I can say that there were faster and slower "days". The day of reinstall the cap was 3MB/s but few days before I had no problem torrenting with full speed (8-11MB/s). The same behaviour I can see in the office. Once it runs so bad that I'm just getting crazy and the other day (say the next one) it runs good enough for me to not to be upset.

So I think it is always a case where a lot of connections are created. But I think it is tens of connections, not thousands.

Please note that we are talking about 3 years old installation of Ubuntu 18.04 upgraded to Ubuntu 20.04 meanwhile. And about 2 days old windows installation where I have firefox and steam... All Microsoft bullshits (e.g. onedrive client) were removed using https://github.com/farag2/Windows-10-Sophia-Script

[northys]

Also I can say that the Windows ISO was downloading with speed ranging from 300-1500kB/s. It was like half time at 300 and half time at 1.5M/s. When I created the final working USB stick using microsoft's tool on my fathers Windows laptop (cheap 500€ laptop, no portmaster) it downloaded and copied the files to the stick under 15 minutes...

[dhaavi]

Also I can say that there were faster and slower "days". The day of reinstall the cap was 3MB/s but few days before I had no problem torrenting with full speed (8-11MB/s). The same behaviour I can see in the office. Once it runs so bad that I'm just getting crazy and the other day (say the next one) it runs good enough for me to not to be upset.

This is really interesting, and also makes this a lot more tricky.

So I think it is always a case where a lot of connections are created. But I think it is tens of connections, not thousands.

We have produced this behavior with a high-intensity nmap scan, so we were well above thousands of connections, so I don't think that tens of connections pose any problems. If that were the case we would be seeing a lot more issues. Also, as soon as the scan stopped, everything returned back to normal almost immediately.

---

I am currently seeing weird IPv6/UDP performance with my virtual Windows testing machine. Will try with a native system within the next days. Do you have IPv6 enabled? If so, please try turning it off for a while.

[northys]

I am currently seeing weird IPv6/UDP performance with my virtual Windows testing machine. Will try with a native system within the next days. Do you have IPv6 enabled? If so, please try turning it off for a while.

I have not disabled it. But my home networking and also the network of my ISP is IPv4 only. Should I still try to disable it?

[dhaavi]

I have not disabled it. But my home networking and also the network of my ISP is IPv4 only. Should I still try to disable it?

No, in that case it does not matter.

[ppacher]

Hi,

we just release portmaster v0.5.7 with support for simple packet metrics (see https://github.com/safing/portmaster/commit/7a83e772f40994cdf43dd6b36c9f4ea02ca5b35d). @northys could you try to enable the metrics on your system? It just creates a simple CSV file containing the number of microseconds and the decision for each packet handled. The next time you encounter the slow throughput issue we will hopefully see an increase in those metrics too. This will at least help use to track down the cause of the issue.

To enable the metrics just add --write-packet-metrics <file-path> to PORTMASTER_ARGS as described here. It's basically the same procedure you already did for the --experimental-nfqueue (which is the default since 0.5.7 now so you can remove that one ;); see https://github.com/safing/portmaster/commit/bdcfc8c566b1c784797c4d4cc9d2d7d41badc8e8 )

The Portmaster will then write new metrics every few seconds to the file specified. If you restart Portmaster the file will be truncated and overwritten! So if you encounter the issue again it would be great if you could copy it away and send it to us. <file-path> can be anything but I'd recommend something like /var/lib/portmaster/metrics.log or so.

You should receive the update to v0.5.7 within the next hour or so.

[northys]

Hi @ppacher,

how can I set it for windows? There is no edit in Services menu.

[dhaavi]

On Windows, you can start the Portmaster with custom parameters using a privileged cmd window:

; stop the service
sc stop PortmasterCore

; start Portmaster with custom flags
C:\ProgramData\Safing\Portmaster\updates\windows_amd64\core\portmaster-core_v0-5-7.exe --data C:\ProgramData\Safing\Portmaster --write-packet-metrics C:\ProgramData\Safing\Portmaster\packet-metrics.log
; use CTRL+C to stop the process

; when done, start service again
sc start PortmasterCore

[northys]

https://bin.privacytools.io/?78292089239e87d2#ILS1x04KCYWYrwEUlcfBSVKx8CV0Vd6BuKq1MPs6A4Q=

[northys]

I downloaded ubuntu while portmaster was running and it went around 11MB/s which is link capacity. I opened Github to report you this and got unable to connect but the ubntu iso was still downloading. So it is definitely something with new connections not the already active ones.

[northys]

I think it is DNS which is not working... Here you can see ping running while Firefox reports "Unable to resolve domain".

[northys]

Yesteeday after I broke portmaster during steam downloading for a hour a two (I played mafia and Steam was opened so I don't know if it downloaded anything in the background) the internet was so unstable that I got DNS page resolved every a minute for a few seconds and then it didnt work again. I had to reboot windows and kill steam immediately after Windows start so I could send you those results with no problems.

e.g. I refreahed this page successfuly but immediate image upload ended with resolution error. upload retry didnt work...

[dhaavi]

Thank you for all this great information - the video was also very helpful.

We had a look at the logs you sent us generated by the --write-packet-metrics flag. It shows similar behavior that we were able to produce. 🎉 While it is likely, this does not mean that this stems from the same problem. We will continue to investigate this problem and hope to have a fix for this soon.

We believe the DNS issues to be related, as when the problem occurs, all network activity is slowed drastically, prohibiting any new network connections. This would also prohibit you from successfully opening the App/UI.

What would further help us, is if you could generate a trace level log while reproducing the issue with Steam. Here is how you can do this using a privileged cmd window:

; stop the service
sc stop PortmasterCore

; start Portmaster with custom flags
C:\ProgramData\Safing\Portmaster\updates\windows_amd64\core\portmaster-core_v0-5-7.exe --data C:\ProgramData\Safing\Portmaster --log trace --write-packet-metrics C:\ProgramData\Safing\Portmaster\packet-metrics.log
; use CTRL+C to stop the process

; when done, start service again
sc start PortmasterCore

Please supply us with the trace log file, as well as the metrics log for referencing the timestamps.

We will try to exactly reproduce with a similar as possible setup in the meanwhile.

[northys]

What would further help us, is if you could generate a trace level log while reproducing the issue with Steam. Here is how you can do this using a privileged cmd window:

I think I'll get to this in the evening.

We will try to exactly reproduce with a similar as possible setup in the meanwhile.

So you are going to buy Mafia: definitive edition? :heart_eyes:

[northys]

For some reason it always boots to 0.5.6 even when I have 0.5.7 installed. For some reason the 0.5.7 wasn't autocompleted by powershell so I almost disturbed @davegson on signal to help with downloading 0.5.7 manually... weird....

The sc stop PortmasterCore does nothing. It won't even say anything. What a fucking OS. Anyway, the notifier and start is running even when stopped manually in Services menu so I couldn't delete logs without killing it from task manager.

And finally - steam won't start even when I see multiple queries to their Frankfurt cache edges.

Hmm, interesting. Firefox can access internet but Steam can't. Sometimes even windows net indicator says "connected, no internet". But I was unable to start steam... I don't think it's company network restriction. Comate often plays games here.

[northys]

https://bin.privacytools.io/?d94eb65b2e68f4c2#jKEvz78T5qDzgndVtEljuJsV71b7cltnRCMnFV7jmCw=

This test took me around 15minutes but portmaster did not write log file even when I tried to run it to check if it creates it before this test and it did... So I deleted the log file and started it again... Loosing my mind with this OS, srsly. Why it won't say fucking permission denied omg.

EDIT: its because portmaster start writes to the log file -.-

Here you can see at least something...

[northys]

It was much easier to break it on my friends 500Mbps FTTH.

Have fun! https://bin.privacytools.io/?4fe3e833feea2c27#GU5EneddPzQ+ELSCFTU11TsDZudMDEGUGzqQ1knb2LM=

[northys]

btw most of those screens have description as their filename so plase hover cursor over them to see it. i'm too lazy to write it again above it =)

[dhaavi]

I was able to reproduce this issue today in a similar environment. 🎉 (Unfortunately not with Mafia: definitive edition)

https://bin.privacytools.io/?d94eb65b2e68f4c2#jKEvz78T5qDzgndVtEljuJsV71b7cltnRCMnFV7jmCw=

I can confirm I am seeing very high packet response rates here, so there is something wrong here.

EDIT: its because portmaster start writes to the log file -.-

Yup. That is the case. There might be an option for writing log files directly in the future. Let's see.

Have fun! https://bin.privacytools.io/?4fe3e833feea2c27#GU5EneddPzQ+ELSCFTU11TsDZudMDEGUGzqQ1knb2LM=

I also see a couple high response times in here. This might be the issue, but I'm not sure:

When I reproduced the issue I did not have these spikes, probably because I turned off process detection in order to see if that was causing trouble. All the response times I saw were very good.

This also makes sense because as far as I can tell at the moment, Steam seems to be the only thing being affected. Or do we have any other confirmed issues for this? @ErichDonGubler vaguely mentioned conferencing systems, but we don't have any more leads on that. There could be a different cause there.

Anyway, Steam downloads were still dying. In my case I updated a game outside of the permitted download time setting. This meant that Steam opened the "Play Game" popup that had the download progress too. This then showed an error - I think it was Content Servers Unreachable. I talked with @ppacher about that and I will test again tomorrow and see if I see anything weird on the network, as the logs seemed perfectly ok.

If you have time to do another quick test, please enable Dev Mode and disable Process Detection in the settings. Be sure to first enable Dev Mode, as else the App will be cut off from the Core. Then please test if the issue persists - it should. If not, this will just get a lot weirder.

[northys]

Unfortunately not with Mafia: definitive edition

I'll buy you a copy if you finish SPN before x-mas :P

This also makes sense because as far as I can tell at the moment, Steam seems to be the only thing being affected. Or do we have any other confirmed issues for this? @ErichDonGubler vaguely mentioned conferencing systems, but we don't have any more leads on that. There could be a different cause there.

Steam is the only one where I'm 100% sure how to reproduce the issue. But still there's there case where the internet was unstable even after steam close. We are talking about 5minutes+ while I tried to post logs and screens here.

@northys - internet was so unstable that I got DNS page resolved every a minute for a few seconds and then it didnt work again

I'm 90% sure that I've tried multiple apps to see wether they do have access to the internet or not. I'm 93.1415% sure that I also turned off steam since I already knew it is the app that starts (I mean "begin" not "causes") the problem. My theory is actually that something gets bloated (let's call it a Q) and it never gets emptied. It's like only 10% of the Q get's processed and the rest of it smells there forever. That would explain why the first "peak" in steam downloads lasts more than any next peak.

@northys - For some reason it always boots to 0.5.6 even when I have 0.5.7 installed. For some reason the 0.5.7 wasn't autocompleted by powershell so I almost disturbed @davegson on signal to help with downloading 0.5.7 manually... weird....

What about this one? I think there is problem that portmaster can't restart itself on Windows and I bet there is a process which rewrites version to use in config or something like this... So I always start with 0.5.6 and it says "update available" - it's already downloaded for few days.

---

If you have time to do another quick test, please enable Dev Mode and disable Process Detection in the settings. Be sure to first enable Dev Mode, as else the App will be cut off from the Core. Then please test if the issue persists - it should. If not, this will just get a lot weirder.

If I'll be able to connect to 500Mbps again I'll do it asap. Othersiwe we will have to wait until friday before I go to home. Hotel and office wifi sux. Yesterday I measured it connected via ethernet cable.

[northys]

1) first time steam did not start https://bin.privacytools.io/?d0fc2c095995819e#M7n2MT/XzxjCa8+qPPX1yN/iUEzdLq4CQQhvKAADbtk=

2) windows reboot https://bin.privacytools.io/?e984bb5660c867e2#6A+e6solI74MlAkRdwtn+K8GwWmzcB1AKM+/ZMg+R/k=

packet metrics https://bin.privacytools.io/?6b15187737798ee5#PKGGtC6BGU3YqN/NIoJq5orHpzRylZ/MbvI0SSoiS5U=

[dhaavi]

Thanks, this confirms my suspicion that this has nothing to do with the high response times to packets, because everything was blazing fast from that perspective now.

I had another test and looked at the network traffic with Wireshark. I made a very interesting discovery: Steam at some point starts to send packets with an unusual high size of about 10KB each. We weren't yet able to prove that this is the cause of the troubles, but considering everything else it seems likely. After discussing the findings with @ppacher, I will have to run the test again to look out for special things and save the results for deeper investigation.

We are currently suspecting that there is a problem with the traffic handling in the Portmaster Kernel Extension on Windows, triggered by unusual traffic from Steam.

If you have a fast proxy available, you could try if using a proxy with Steam works around the issue, as this would normalize the traffic before the Portmaster would handle it. I don't know yet when we will have time to continue this investigation, as there is a lot of stuff going on currently and we have a very important internal deadline to meet this month.

[northys]

Meh, I'll continue testing portmaster on Windows when this gets resolved.

Btw there are still many unanswered questions and potential bugs / feature requests in this thread.

[northys]

What about this? Logged during shutdown.

201012 22:42:51.210 xt/handler:048 > WARN 097 failed to get packet from windows kext: the windows kernel extension (driver) is not ready to accept commands

[dhaavi]

Meh, I'll continue testing portmaster on Windows when this gets resolved.

Ok. I hope to find time soon. There are some very exciting updates coming! (Which need to be prioritized due to internal constraints.)

Btw there are still many unanswered questions and potential bugs / feature requests in this thread.

I'm sorry I missed them. I often get my head tangled up in the main issue that I then forget answering. It would be best to create separate issues for separate problems, that makes it easy to keep track of them.

What about this? Logged during shutdown. 201012 22:42:51.210 xt/handler:048 > WARN 097 failed to get packet from windows kext: the windows kernel extension (driver) is not ready to accept commands

This is normal during shutdown, but thanks for the hint. These won't show up anymore when normal with the next update.

For some reason it always boots to 0.5.6 even when I have 0.5.7 installed.

This indeed should not happen. Maybe permissions were screwed up during manual testing. You could try removing v0.5.7 from the updates directory and have the Portmaster re-download it. If that does not fix it, please open an issue.

---

Anything else I missed?

[northys]

For clarification: Did you experience Portmaster slowing down the network on both Linux and Windows? Or was it Windows only?

Both. The networking was ridicuously slow on my Ubuntu before reinstall. It took me 1.5h to download the 5G Windows ISO from Microsoft servers. Fedora ISO was much faster but it didn't go over 3MB/

I do have troubles with networking stability on fedora. I think the problem is caused by docker but I'm networking noob so I can't debug it any further. Portmaster is not running on my laptop currently. Either on windows or on fedora.

So let's say this problem is Windows only.

[dhaavi]

I wanted to do some further testing yesterday, but to my surprise I was not able to reproduce the issue.

It could be that either a recent Windows 10 or Steam update fixed the issue on their side. If you find time, please check if the issue still persists for you.

If it does, I would ask you to take a packet capture of the dying connection. I really want to get down on this, as I don't want a possible bug sitting around, not knowing how to trigger it.

For the packet capture, please install Wireshark (https://www.wireshark.org/#download) and let it capture packets on the interface you use to connect to the Internet. If you have the capture, we can work out how to best extract only the necessary information and how to transfer it.

[northys]

@dhaavi what windows and portmaster version have you tried with? just to confurm we have the same env

[dhaavi]

My versions are:

Windows 10 Pro: 1909, 18363.1139
Steam: Oct 6 2020, 1602115886
Portmaster: v0.5.7

[northys]

Meh, bad private pastebin. New one with 1 week expiration.

https://bin.privacytools.io/?5fbeb14e3501d1bd#N7dhKDk3/gmS8rFNMn8ApcPvOm+1HKzY21RflIrGKkc= https://bin.privacytools.io/?7b77c06ea0020675#xpmI7rp5Ftp5FGHkpAwmL96cnbm4iARZLwEt45BQaQ0=

[northys]

Windows 10 Home 19041.572 Steam 1603487799 Portmaster v0.5.7

both windows and steam is piece of shit and didnt let me to copy the version number so maybe I typed it wrong...

the behaviour was same as the one originaly reported

[dhaavi]

Thanks for taking the time to check the issue again. I will try to reproduce the behavior in the coming weeks.

[dhaavi]

I've been been able to reproduce the issue again and also found underlying problem. Apparently, DNS requests slipped by the Portmaster and their response was then blocked, because believed that they were incoming requests.

After a couple bug fixes in the Portmaster Kext this issue should be resolved now. I've still seen some interesting behavior, but only with IPv6 - I will continue to investigate.

If you find the time, it would be great if you could see if the fix works for you too. The relevant version are v0.6.3 of the core and v1.0.7 of the kext. Also, I think the UI changed a bit since the last time you used the Portmaster. ;)

[northys]

I've been been able to reproduce the issue again and also found underlying problem. Apparently, DNS requests slipped by the Portmaster and their response was then blocked, because believed that they were incoming requests.

I can confirm the same weird behaviour after installing Portmaster again. I have only Firefox and Telegram open. I have already disabled Steam on startup because I was getting crazy from all the windows just popping up randomly at random positions taking the focus. It just instantly returned couldn't resolve google.com, then root.cz and everything new that wasn't cached already failed to load. I could browse GitHub and Netflix with no issues. Both pages were opened before the first "trigger" of the issue. It works now after a minute or so. Same behaviour, steam not required to reproduce it. Even high load is not necessary.

Also, I think the UI changed a bit since the last time you used the Portmaster. ;)

Nice

[northys]

Did I mention that I've run https://github.com/farag2/Windows-10-Sophia-Script ?

[dhaavi]

I can confirm the same weird behaviour after installing Portmaster again.

Can you confirm that Steam received DNS replies that were blocked?

It just instantly returned couldn't resolve google.com, then root.cz and everything new that wasn't cached already failed to load. I could browse GitHub and Netflix with no issues. Both pages were opened before the first "trigger" of the issue. It works now after a minute or so. Same behaviour, steam not required to reproduce it. Even high load is not necessary.

This sound more like a DNS issue - which seems to be working fine now for everyone. Did you remove all data when uninstalling? If not, clearing the DNS cache in the Settings should do the trick.

Did I mention that I've run https://github.com/farag2/Windows-10-Sophia-Script ?

I looked into it and I don't think this would trigger any problems.

[northys]

I've decided that I'll start playing wow and I've already successfully forgotten about me trying the latest version already so I:

• installed portmaster again, I did not purge anything
• rebooted from my dualboot to windows after 6th try (grub is on laptop screen which is closed and I'm too lazy to open it to see which line is windows)
• downloaded battle.net
• went afk while wow was downloading for 30 minutes. it went full speed through my 2.4GHz Wi-Fi even when I was playing music from my phone on my bluetooth speaker. this causes interefences on fedora and produces a huge packet loss. don't know why. I mention this just to say everything was smooth untill
• friend has sent me a tweet link (twitter.com) which opened with no problem. today i havent had twitter opened on windows yet so it couldnt be cached
• this tweet contained link to another tweet leading to t.com
• did not resolve. server not couldnt be found.
• ctrl + f5 on twitter.com - server couldnt be found
• ctrl + f5 on battle.net - server couldnt be found
• pinged 8.8.8.8 - no problem
• ping google.com - server not found after 60s timeout
• battle.net is still downloading wow at full speed 10M/s via P2P

I'm after update to 20H2 or how the hell they called it.

[dhaavi]

Great! So that sounds like the download issue has been finally resolved. Please confirm.

I think it's best we continue working on the DNS issue over in the other issue you opened: #132. Maybe start off by posting your current configuration there, especially which DNS servers you are using.

[northys]

@dhaavi I reinstalled portmaster, took care about the "make sure to remove all portmaster data before reinstalling portmaster" step and I'm happy to announce that I wasn't able to reproduce the issue mentioned before.

• steam startups with no issue
• reinstalled 35G mafia (steam) with no issue, played wow and runned dig google.com in loop during download
• took about an hour to finish and I found no issue while eye checking everything every 1 minute or so

before testing I've connected my laptop to home wifi using my old router because it should have better anntena than the one buildin in display. not my connection is more stable than ever.

[northys]

FML. I'm writing a comment about how beautiful everything is and submit results in "Server not found" :D :D :D

This sound more like a DNS issue - which seems to be working fine now for everyone. Did you remove all data when uninstalling? If not, clearing the DNS cache in the Settings should do the trick.

So what now? I'm getting crazy debugging portmaster on bullshitOS. It's like every part of this OS was perfectly crafted NOT to be normal.