laravel / valet

A more enjoyable local development experience for Mac.
https://laravel.com/docs/valet
MIT License
2.49k stars 691 forks source link

Running Valet with ExpressVPN not work #1441

Closed Recrus closed 8 months ago

Recrus commented 10 months ago

Description

This issue has been discussed before. But at that time there was a fairly simple solution, which, unfortunately, became unavailable with the new version of ExpressVPN. The bottom line is that when you start vpn and valet at the same time, the valet server stops responding. This used to be solved by disabling the "DNS: Only use ExpressVPN DNS servers while connected" feature in the vpn app, but this feature has now been removed.

*I use Macbook Air on m1 chip

Steps To Reproduce

  1. Install ExpressVPN and turn it on
  2. Use valet link and try to send request to your server

Diagnosis

sw_vers
ProductName:       macOS
ProductVersion:     13.4.1
ProductVersionExtra:    (c)
BuildVersion:       22F770820d
valet --version
Laravel Valet 4.0.2
cat ~/.config/valet/config.json
{
    "tld": "test",
    "loopback": "127.0.0.1",
    "paths": [
        "/Users/_recrus/.config/valet/Sites"
    ]
}
cat ~/.composer/composer.json
{
    "require": {
        "laravel/valet": "^4.0"
    }
}
composer global diagnose
Changed current directory to /Users/_recrus/.composer
Checking composer.json: WARNING
No license specified, it is recommended to do so. For closed-source software you may use "proprietary" as license.
Checking platform settings: OK
Checking git settings: OK git version 2.39.2
Checking http connectivity to packagist: OK
Checking https connectivity to packagist: OK
Checking github.com rate limit: OK
Checking disk free space: OK
Checking pubkeys: 
Tags Public Key Fingerprint: 57815BA2 7E54DC31 7ECC7CC5 573090D0  87719BA6 8F3BB723 4E5D42D0 84A14642
Dev Public Key Fingerprint: 4AC45767 E5EC2265 2F0C1167 CBBB8A2B  0C708369 153E328C AD90147D AFE50952
OK
Checking composer version: OK
Composer version: 2.6.2
PHP version: 8.2.4
PHP binary path: /opt/homebrew/Cellar/php/8.2.4/bin/php
OpenSSL version: OpenSSL 1.1.1t  7 Feb 2023
cURL version: 8.0.1 libz 1.2.11 ssl (SecureTransport) OpenSSL/1.1.1t
zip: extension present, unzip present, 7-Zip not available
composer global outdated
Changed current directory to /Users/_recrus/.composer
Info from https://repo.packagist.org: #StandWithUkraine
Legend:
! patch or minor release available - update recommended
~ major release available - update possible

Direct dependencies required in composer.json:
laravel/valet                      v4.0.2  ! v4.1.4   A more enjoyable local...

Transitive dependencies not required in composer.json:
guzzlehttp/guzzle                  7.5.1   ! 7.8.0    Guzzle is a PHP HTTP c...
guzzlehttp/promises                1.5.2   ~ 2.0.1    Guzzle promises library
guzzlehttp/psr7                    2.5.0   ! 2.6.1    PSR-7 message implemen...
illuminate/collections             v10.9.0 ! v10.21.0 The Illuminate Collect...
illuminate/conditionable           v10.9.0 ! v10.21.0 The Illuminate Conditi...
illuminate/container               v10.9.0 ! v10.21.0 The Illuminate Contain...
illuminate/contracts               v10.9.0 ! v10.21.0 The Illuminate Contrac...
illuminate/macroable               v10.9.0 ! v10.21.0 The Illuminate Macroab...
mnapoli/silly                      1.8.1   ! 1.8.3    Silly CLI micro-framew...
symfony/console                    v6.2.10 ! v6.3.4   Eases the creation of ...
symfony/deprecation-contracts      v3.2.1  ! v3.3.0   A generic function and...
symfony/event-dispatcher           v6.2.8  ! v6.3.2   Provides tools that al...
symfony/event-dispatcher-contracts v3.2.1  ! v3.3.0   Generic abstractions r...
symfony/polyfill-ctype             v1.27.0 ! v1.28.0  Symfony polyfill for c...
symfony/polyfill-intl-grapheme     v1.27.0 ! v1.28.0  Symfony polyfill for i...
symfony/polyfill-intl-normalizer   v1.27.0 ! v1.28.0  Symfony polyfill for i...
symfony/polyfill-mbstring          v1.27.0 ! v1.28.0  Symfony polyfill for t...
symfony/process                    v6.2.10 ! v6.3.4   Executes commands in s...
symfony/service-contracts          v3.2.1  ! v3.3.0   Generic abstractions r...
symfony/string                     v6.2.8  ! v6.3.2   Provides an object-ori...
ls -al /etc/sudoers.d/
total 0
drwxr-xr-x   2 root  wheel    64 Jun 15 13:08 .
drwxr-xr-x  82 root  wheel  2624 Aug  6 12:36 ..
brew config
HOMEBREW_VERSION: 4.1.7
ORIGIN: https://github.com/Homebrew/brew
HEAD: d4444b563e24ac7c05a93121c464c02dfa04d44f
Last commit: 7 days ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 2346e448ef3dfc950950371e8775603cd6bdb25c
Core tap last commit: 56 minutes ago
Core tap branch: master
Core tap JSON: 03 Sep 21:08 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 8
Homebrew Ruby: 2.6.10 => /opt/homebrew/Library/Homebrew/vendor/portable-ruby/2.6.10_1/bin/ruby
CPU: octa-core 64-bit arm_firestorm_icestorm
Clang: 14.0.3 build 1403
Git: 2.39.2 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 7.88.1 => /usr/bin/curl
macOS: 13.4.1-arm64
CLT: 14.3.1.0.1.1683849156
Xcode: N/A
Rosetta 2: false
brew services list
Name    Status User    File
dnsmasq none            root    
nginx   none            root    
php     none            root    
php@7.4 error  6        _recrus ~/Library/LaunchAgents/homebrew.mxcl.php@7.4.plist
brew list --formula --versions | grep -E "(php|nginx|dnsmasq|mariadb|mysql|mailhog|openssl)(@\d\..*)?\s"
dnsmasq 2.89
nginx 1.23.4
openssl@1.1 1.1.1t
php 8.2.4
php@7.4 7.4.30_1
brew outdated
aom
apr
apr-util
brotli
ca-certificates
curl
freetds
freetype
gd
highway
icu4c
imath
jpeg-turbo
jpeg-xl
krb5
libavif
libnghttp2
libpng
libpq
libssh2
libtiff
libzip
little-cms2
nginx
openexr
openldap
openssl@1.1
php
php@7.4
python3
rtmpdump
sqlite
unixodbc
webp
xz
zstd
brew tap
homebrew/cask
homebrew/core
homebrew/services
php -v
PHP 8.2.4 (cli) (built: Mar 16 2023 16:10:27) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.2.4, Copyright (c) Zend Technologies
    with Zend OPcache v8.2.4, Copyright (c), by Zend Technologies
which -a php
/opt/homebrew/Cellar/php/8.2.4/bin/php
/opt/homebrew/bin/php
/opt/homebrew/bin/php
php --ini
Configuration File (php.ini) Path: /opt/homebrew/etc/php/8.2
Loaded Configuration File:         /opt/homebrew/etc/php/8.2/php.ini
Scan for additional .ini files in: /opt/homebrew/etc/php/8.2/conf.d
Additional .ini files parsed:      /opt/homebrew/etc/php/8.2/conf.d/error_log.ini,
/opt/homebrew/etc/php/8.2/conf.d/ext-opcache.ini,
/opt/homebrew/etc/php/8.2/conf.d/php-memory-limits.ini
nginx -v
nginx version: nginx/1.23.4
curl --version
curl 7.88.1 (x86_64-apple-darwin22.0) libcurl/7.88.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.51.0
Release-Date: 2023-02-20
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe UnixSockets
php --ri curl
curl

cURL support => enabled
cURL Information => 8.0.1
Age => 10
Features
AsynchDNS => Yes
CharConv => No
Debug => No
GSS-Negotiate => No
IDN => Yes
IPv6 => Yes
krb4 => No
Largefile => Yes
libz => Yes
NTLM => Yes
NTLMWB => Yes
SPNEGO => Yes
SSL => Yes
SSPI => No
TLS-SRP => Yes
HTTP2 => Yes
GSSAPI => Yes
KERBEROS5 => Yes
UNIX_SOCKETS => Yes
PSL => No
HTTPS_PROXY => Yes
MULTI_SSL => Yes
BROTLI => Yes
ALTSVC => Yes
HTTP3 => No
UNICODE => No
ZSTD => Yes
HSTS => Yes
GSASL => No
Protocols => dict, file, ftp, ftps, gopher, gophers, http, https, imap, imaps, ldap, ldaps, mqtt, pop3, pop3s, rtmp, rtmpe, rtmps, rtmpt, rtmpte, rtmpts, rtsp, scp, sftp, smb, smbs, smtp, smtps, telnet, tftp
Host => aarch64-apple-darwin21.6.0
SSL Version => (SecureTransport) OpenSSL/1.1.1t
ZLib Version => 1.2.11
libSSH Version => libssh2/1.10.0

Directive => Local Value => Master Value
curl.cainfo => no value => no value
/opt/homebrew/bin/ngrok version
sudo: /opt/homebrew/bin/ngrok: command not found
ls -al ~/.ngrok2
ls: /Users/_recrus/.ngrok2: No such file or directory
brew info nginx
==> nginx: stable 1.25.2 (bottled), HEAD
HTTP(S) server and reverse proxy, and IMAP/POP3 proxy server
https://nginx.org/
/opt/homebrew/Cellar/nginx/1.23.4 (26 files, 2.2MB) *
  Poured from bottle using the formulae.brew.sh API on 2023-04-30 at 20:07:33
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/n/nginx.rb
License: BSD-2-Clause
==> Dependencies
Required: openssl@3, pcre2
==> Options
--HEAD
    Install HEAD version
==> Caveats
Docroot is: /opt/homebrew/var/www

The default port has been set in /opt/homebrew/etc/nginx/nginx.conf to 8080 so that
nginx can run without sudo.

nginx will load all files in /opt/homebrew/etc/nginx/servers/.

To start nginx now and restart at login:
  brew services start nginx
Or, if you don't want/need a background service you can just run:
  /opt/homebrew/opt/nginx/bin/nginx -g daemon\ off\;
==> Analytics
install: 18,805 (30 days), 56,674 (90 days), 100,855 (365 days)
install-on-request: 18,777 (30 days), 56,580 (90 days), 100,707 (365 days)
build-error: 4 (30 days)
brew info php
==> php: stable 8.2.10 (bottled), HEAD
General-purpose scripting language
https://www.php.net/
/opt/homebrew/Cellar/php/8.2.4 (520 files, 83.3MB) *
  Poured from bottle using the formulae.brew.sh API on 2023-04-02 at 00:20:22
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/p/php.rb
License: PHP-3.01
==> Dependencies
Build: httpd, pkg-config
Required: apr, apr-util, argon2, aspell, autoconf, curl, freetds, gd, gettext, gmp, icu4c, krb5, libpq, libsodium, libzip, oniguruma, openldap, openssl@3, pcre2, sqlite, tidy-html5, unixodbc
==> Options
--HEAD
    Install HEAD version
==> Caveats
To enable PHP in Apache add the following to httpd.conf and restart Apache:
    LoadModule php_module /opt/homebrew/opt/php/lib/httpd/modules/libphp.so

    
        SetHandler application/x-httpd-php
    

Finally, check DirectoryIndex includes index.php
    DirectoryIndex index.php index.html

The php.ini and php-fpm.ini file can be found in:
    /opt/homebrew/etc/php/8.2/

To start php now and restart at login:
  brew services start php
Or, if you don't want/need a background service you can just run:
  /opt/homebrew/opt/php/sbin/php-fpm --nodaemonize
==> Analytics
install: 55,116 (30 days), 160,436 (90 days), 286,505 (365 days)
install-on-request: 51,246 (30 days), 148,932 (90 days), 265,642 (365 days)
build-error: 44 (30 days)
brew info openssl
==> openssl@3: stable 3.1.2 (bottled)
Cryptography and SSL/TLS Toolkit
https://openssl.org/
Not installed
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/o/openssl@3.rb
License: Apache-2.0
==> Dependencies
Required: ca-certificates
==> Caveats
A CA file has been bootstrapped using certificates from the system
keychain. To add additional certificates, place .pem files in
  /opt/homebrew/etc/openssl@3/certs

and run
  /opt/homebrew/opt/openssl@3/bin/c_rehash
==> Analytics
install: 409,822 (30 days), 1,058,225 (90 days), 1,439,511 (365 days)
install-on-request: 40,302 (30 days), 167,887 (90 days), 285,035 (365 days)
build-error: 1,561 (30 days)
openssl version -a
LibreSSL 3.3.6
built on: date not available
platform: information not available
options:  bn(64,64) rc4(ptr,int) des(idx,cisc,16,int) blowfish(idx) 
compiler: information not available
OPENSSLDIR: "/private/etc/ssl"
openssl ciphers
AEAD-CHACHA20-POLY1305-SHA256:AEAD-AES256-GCM-SHA384:AEAD-AES128-GCM-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:GOST2012256-GOST89-GOST89:DHE-RSA-CAMELLIA256-SHA256:DHE-RSA-CAMELLIA256-SHA:GOST2001-GOST89-GOST89:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:CAMELLIA256-SHA256:CAMELLIA256-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-CAMELLIA128-SHA256:DHE-RSA-CAMELLIA128-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:CAMELLIA128-SHA256:CAMELLIA128-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:DES-CBC3-SHA
sudo nginx -t
nginx: the configuration file /opt/homebrew/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /opt/homebrew/etc/nginx/nginx.conf test is successful
which -a php-fpm
/opt/homebrew/sbin/php-fpm
/opt/homebrew/opt/php/sbin/php-fpm -v
PHP 8.2.4 (fpm-fcgi) (built: Mar 16 2023 16:10:28)
Copyright (c) The PHP Group
Zend Engine v4.2.4, Copyright (c) Zend Technologies
    with Zend OPcache v8.2.4, Copyright (c), by Zend Technologies
sudo /opt/homebrew/opt/php/sbin/php-fpm -y /opt/homebrew/etc/php/8.2/php-fpm.conf --test
[04-Sep-2023 00:09:06] NOTICE: configuration file /opt/homebrew/etc/php/8.2/php-fpm.conf test is successful
ls -al ~/Library/LaunchAgents | grep homebrew
-rw-r--r--   1 _recrus  staff   789 Sep 20  2022 homebrew.mxcl.php@7.4.plist
ls -al /Library/LaunchAgents | grep homebrew

ls -al /Library/LaunchDaemons | grep homebrew
-rw-r--r--   1 root  admin   797 May  2 15:13 homebrew.mxcl.dnsmasq.plist
-rw-r--r--   1 root  admin   685 Sep  3 23:36 homebrew.mxcl.nginx.plist
-rw-r--r--   1 root  admin   781 May  2 15:13 homebrew.mxcl.php.plist
ls -al /Library/LaunchDaemons | grep "com.laravel.valet."

ls -aln /etc/resolv.conf
lrwxr-xr-x  1 0  0  22 Jun 15 13:08 /etc/resolv.conf -> ../var/run/resolv.conf
cat /etc/resolv.conf
#
# macOS Notice
#
# This file is not consulted for DNS hostname resolution, address
# resolution, or the DNS query routing mechanism used by most
# processes on this system.
#
# To view the DNS configuration used by this system, use:
#   scutil --dns
#
# SEE ALSO
#   dns-sd(1), scutil(8)
#
# This file is automatically generated.
#
search expressvpn
nameserver 100.64.100.1
ifconfig lo0
lo0: flags=8049 mtu 16384
    options=1203
    inet 127.0.0.1 netmask 0xff000000 
    inet6 ::1 prefixlen 128 
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
    nd6 options=201
sh -c 'echo "------\n/opt/homebrew/etc/nginx/valet/valet.conf\n---\n"; cat /opt/homebrew/etc/nginx/valet/valet.conf | grep -n "# valet loopback"; echo "\n------\n"'
------
/opt/homebrew/etc/nginx/valet/valet.conf
---

3:    #listen VALET_LOOPBACK:80; # valet loopback

------
sh -c 'for file in ~/.config/valet/dnsmasq.d/*; do echo "------\n~/.config/valet/dnsmasq.d/$(basename $file)\n---\n"; cat $file; echo "\n------\n"; done'
------
~/.config/valet/dnsmasq.d/tld-test.conf
---

address=/.test/127.0.0.1
listen-address=127.0.0.1

------
sh -c 'for file in ~/.config/valet/nginx/*; do echo "------\n~/.config/valet/nginx/$(basename $file)\n---\n"; cat $file | grep -n "# valet loopback"; echo "\n------\n"; done'
------
~/.config/valet/nginx/*
---

cat: /Users/_recrus/.config/valet/nginx/*: No such file or directory

------
drbyte commented 10 months ago

When you run the ExpressVPN client directly on your Mac, it interrupts the Mac's already-configured DNS functionality, both for its own internal operations and for Valet's operations via dnsmasq.

If they've removed all configuration settings to allow local DNS serving, then you may have limited options.

I have 2 suggestions:

  1. Contact their tech support (they have a Live Chat) and ask for instructions on how to allow local DNS lookups when the Express VPN client is active. Be sure to tell them you're using dnsmasq on 127.0.0.1 to serve "*.test" domains from 127.0.0.1 / localhost. Post back with whatever solution they give you. And perhaps encourage them to post an FAQ for it on their website.

  2. Alternatively, you could configure your Router to use your ExpressVPN subscription. This would cause ALL internet activity to go through ExpressVPN, (for everyone using your router). By removing ExpressVPN from your Mac (and putting it on the wifi/router instead) will allow Valet to function normally for all its own internal *.test websites, but all your regular internet activity would go through the router, which would send it through ExpressVPN automatically. Their website shows two ways to do that, depending on the kind of router you have: https://www.expressvpn.com/support/vpn-setup/#manual-router https://www.expressvpn.com/support/vpn-setup/#router-setup

Recrus commented 10 months ago

@drbyte I tried to write to support, since their service does not support connecting my router and because I am afraid that I can only make things worse.

Support didn't give me a solution. "Thanks for that and we're very sorry for the trouble. But we'd like to let you know that currently, when you use ExpressVPN, your DNS requests are handled directly by ExpressVPN, with no exposure to third parties. ExpressVPN runs its own DNS servers and when you are connected to ExpressVPN you automatically use these servers — so no one else can get hold of your information or hijack your connection. Provider.

I'm really sorry, but currently, we don't have an option or feature in the app where you can change the DNS settings. But you may try changing the DNS settings on your device instead and see if it'll work for you."

drbyte commented 10 months ago

I don't have an ExpressVPN subscription (and don't want to start a trial), so I can't directly test the app. However, while inspecting the app itself, I see that it still contains language-strings for advanced settings that control the local network. So, I offer the following...

  1. In the application's Preferences screen, is there an "Advanced Settings" window of some sort? Is there a "DNS" sub section to that window? Can you provide screenshots of the various preferences sections?

    "preferences_screen.advanced.dns_label.text" = "DNS:";
    "preferences_screen.advanced.dns_checkbox.text" = "Only use ExpressVPN DNS servers while connected";
  2. Also, sometimes holding down the OPTION key when looking at application menus will expose "hidden" menu options. Does that do anything in the app's menus?

  3. The app supports multiple protocols: does changing protocols from Automatic to something else solve the problem?

  4. In Preferences, does ticking the box to Allow access to devices on the local network (such as printers or file servers) solve the problem?

  5. Does downgrading to v10 solve the problem? https://www.expressvpn.works/clients/mac/expressvpn_mac_10.3.0.49_release.pkg

  6. What's the output when you run this from the command line:

    defaults read com.expressvpn.ExpressVPN

    (Don't post the hex codes or any UUIDs, as those may be private information, and not needed here. I'm more interested in the settings keys that may be registered.)

  7. Inside the app is a defaults.plist file which has the following set to true. Changing them probably makes no difference, particularly because they're already set to true, which would seem to be desirable.

    <key>useDNSServers</key>
    <true/>
    <key>allowLANTraffic</key>
    <true/>
Recrus commented 10 months ago

@drbyte, I've tried all the options you suggested, but unfortunately, none of them worked. Let me provide more details:

  1. There is no DNS section in the "Advanced Settings." Here are some screenshots:

    Screenshot 2023-09-05 at 23 12 53 Screenshot 2023-09-05 at 23 13 01 Screenshot 2023-09-05 at 23 13 18
  2. Holding the Option key doesn't have any effect.

  3. Changing the protocols doesn't resolve the issue.

  4. The relevant setting is already ticked.

  5. Downgrading is not an option for me; when I install either v10.0 or v10.39, I can't even sign in.

  6. Here are my current settings:

{ "MASPreferences AppLauncherPreferences Frame": "{{0, 0}, {568, 273}}", "MASPreferences Frame Top Left": "{1687, 1233}", "MASPreferences ProtocolPreferences Frame": "{{0, 0}, {568, 406}}", "MASPreferences Selected Identifier View": "GeneralPreferences", "MASPreferences ThreatManagerPreferences Frame": "{{0, 0}, {569, 204}}", "NSWindow Frame LocationPickerWindow": "1981 535 366 605 0 0 3008 1667 ", "NSWindow Frame MainWindow": "1607 535 366 605 0 0 3008 1667 ", "ShouldShowClickToConnectHintKey": 0, "SigninDelayHintMsgKey": 20, "XVActivatationDate": "ANONYMIZED", "XVCurrentIAMDisplayDurationKey": 1800, "XVCurrentIAMDisplayTimeKey": "ANONYMIZED", "XVCurrentIAMIndexKey": 1, "XVFMInstallationId": "ANONYMIZED", "XVLastConnectTimeArray": "ANONYMIZED", "XVPreferencesLaunchOnStartupScreenIsDoneKey": 1, "XVReportingScreenIsDoneKey": 1, "XVVPNBrowserExtensionPromobarAlreadyClicked": 1, "allowLANTraffic": 1, "com.launchDarkly.ConnectionInformationStore.connectionInformationKey": "ANONYMIZED", "com.launchdarkly.DiagnosticCache.diagnosticData.mob-e04a4355-a5f5-46ea-b501-8e0006293fd5": "ANONYMIZED", "enableNetworkLock": 1, "enableXVCA": 1, "hasWarnedUserAboutProtocol": 1, "kExpressVPNLaunchCountKey": 19, "kXVAppLauncherHasBeenShownKey": 1, "launchOnStartup": 0, "ldDeviceIdentifier": "ANONYMIZED", "previousVersion": "11.39.0", "protocol": "auto", "recentLocations": [ "ANONYMIZED", "Smart Location" ] }

  1. Also, I can't locate a "defaults.plist" file within the ExpressVPN.app directory. Only an "Info.plist" file is present, which doesn't include the "useDNSServers" and "allowLANTraffic" keys.
drbyte commented 10 months ago
  1. I can't locate a "defaults.plist" file within the ExpressVPN.app directory

This was where I found it: /Applications/ExpressVPN.app/Contents/Resources/Defaults.plist .. but I'm not sure if "Defaults" means it's used "once" (and thus editing it is pointless), or if it's read regularly. I suspect it's used to set things into the defaults database on the mac registry. And probably only the useDNSServers setting would be relevant ... but might not be what we're looking for anyway.

Unfortunately they dropped the Split Tunneling feature for MacOS 11 and above, else it could be another thing to try. In fact maybe that's what they used behind the scenes to offer the former tickbox that allowed the local bypass.

drbyte commented 10 months ago

One workaround I saw posted on a Linux-related article was to go oldschool and manually edit /etc/hosts and add each of your local domains there, each on a separate line. ie: 127.0.0.1 foo.test That's the grassroots way to handle local DNS. Tools like dnsmasq allow slightly more complex handling via additional services, but ExpressVPN is bypassing those services.

drbyte commented 10 months ago

Question: both before-and-after-connecting to ExpressVPN, what's in your /etc/resolv.conf file? (By default it's basically reflecting whatever's in your MacOS Network System Preferences pane for DNS settings. Valet works best when nameserver 127.0.0.1 is listed there (and put there automatically via the Preferences app). It probably also contains "search lan", which is fine.) I'm not sure whether ExpressVPN changes that file's contents when connected or not.

Recrus commented 10 months ago

One workaround I saw posted on a Linux-related article was to go oldschool and manually edit /etc/hosts and add each of your local domains there, each on a separate line. ie: 127.0.0.1 foo.test That's the grassroots way to handle local DNS. Tools like dnsmasq allow slightly more complex handling via additional services, but ExpressVPN is bypassing those services.

Ok, I see. I will try that. As for 'defaults,' it's a matter of capitalization. I used 'defaults.plist' instead of 'Def...'.

And the contents of my /etc/resolv.conf file remain the same before and after connecting to ExpressVPN. Here is what I see:

#
# macOS Notice
#
# This file is not consulted for DNS hostname resolution, address
# resolution, or the DNS query routing mechanism used by most
# processes on this system.
#
# To view the DNS configuration used by this system, use:
#   scutil --dns
#
# SEE ALSO
#   dns-sd(1), scutil(8)
#
# This file is automatically generated.
#
nameserver MY_IP
Recrus commented 10 months ago

Would it be helpful if I provide the output of the scutil --dns command? This will show the current DNS configuration that my macOS system is actually using.

drbyte commented 10 months ago

Would it be helpful if I provide the output of the scutil --dns command? This will show the current DNS configuration that my macOS system is actually using.

Sure. Can't hurt.

See if it's different when connected too.

Recrus commented 10 months ago

Here is the output from scutil --dns before and after connecting to ExpressVPN:

DNS configuration

resolver #1
  nameserver[0] : [REDACTED]
  if_index : 12 (en0)
  flags    : Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

resolver #8
  domain   : test
  nameserver[0] : 127.0.0.1
  flags    : Request A records, Request AAAA records
  reach    : 0x00030002 (Reachable,Local Address,Directly Reachable Address)

DNS configuration (for scoped queries)

resolver #1
  nameserver[0] : [REDACTED]
  if_index : 12 (en0)
  flags    : Scoped, Request A records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)
DNS configuration

resolver #1
  search domain[0] : expressvpn
  nameserver[0] : [REDACTED]
  flags    : Request A records
  reach    : 0x00000002 (Reachable)

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

resolver #8
  domain   : test
  nameserver[0] : 127.0.0.1
  flags    : Request A records, Request AAAA records
  reach    : 0x00030002 (Reachable,Local Address,Directly Reachable Address)

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : expressvpn
  nameserver[0] : [REDACTED]
  if_index : 12 (en0)
  flags    : Scoped, Request A records
  reach    : 0x00000002 (Reachable)
Recrus commented 10 months ago

@drbyte, I tried manually editing the /etc/hosts file as you suggested, and it did serve as a workaround. My local domains are resolving as expected now. However, this seems more like a temporary fix rather than a complete solution...

Thank you for your help so far!

drbyte commented 10 months ago

When I use ProtonVPN, the output of scutil --dns shows that ProtonVPN added itself after my localhost dnsmasq service:

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : lan
  nameserver[0] : 127.0.0.1
  if_index : 5 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00030002 (Reachable,Local Address,Directly Reachable Address)

resolver #2
  nameserver[0] : 10.1.0.1
  if_index : 14 (ipsec0)
  flags    : Scoped, Request A records
  reach    : 0x00000002 (Reachable)

... and that lets me serve stuff locally.

EDIT: And, while there's a risk that having dnsmasq handling lookups could lead to Dns-Leakage if configured to use 3rd party DNS providers, when I test for leakage when ProtonVPN is activated, I'm seeing no leakage reported.

So ... if we do come up with a way to let dnsmasq still work locally, a VERY important question to ask is: "why" you're using a VPN in the first place. If it's to completely hide all your online activity then DNS-Leakage would be a concern to be diligent about; If the VPN is instead merely to access certain destination hosts without disclosing your own ISP IP to them, then dns leakage is less of a worry point.

drbyte commented 10 months ago

Are there any files in /var/run/expressvpn/config/ directory? If yes, what's in them?

drbyte commented 10 months ago

If you understand a bunch about resolv.conf and networking configurations, this post contains some information that may be interesting to explore, albeit requiring translation to macos instead of unix: https://unix.stackexchange.com/a/688325 (I'm kinda just posting this here for simple future reference when I have more time to dig.)

drbyte commented 10 months ago

And probably only the useDNSServers setting would be relevant

This is probably where that boolean setting could be set: ~/Library/Application Support/com.expressvpn.ExpressVPN/ExpressVPNConfigurationOverrides.plist

Recrus commented 10 months ago

Hello @drbyte,

Apologies for the delayed response.

In exploring /var/run/expressvpn/config/, I came across the following:

Screenshot 2023-09-10 at 22 32 04

At the moment, I haven't delved into the intricacies of resolv.conf; however, I appreciate your suggestion, and I might explore it further at a later time!

If you understand a bunch about resolv.conf and networking configurations..

As for the ExpressVPNConfigurationOverrides.plist file, I find myself a bit uncertain. Currently, my file structure looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict/>
</plist>

I am pondering whether I need to incorporate the following lines into it:

<key>useDNSServers</key>
    <false/>

Could you possibly shed some light on this?

Thank you!

drbyte commented 10 months ago

What's the content of that /var/run/expressvpn/config/he4106649726 file?

It'd be worth exploring whether these changes make any difference to ExpressVPNConfigurationOverrides.plist

<plist version="1.0">
-<dict/>
+<dict>
+   <key>useDNSServers</key>
+   <true/>
+</dict>
</plist>
Recrus commented 10 months ago

I'm facing an issue where the /var/run/expressvpn/config/he4106649726 file is consistently empty. I tried to include updates to ExpressVPNConfigurationOverrides.plist, the issue persists.

drbyte commented 10 months ago

Okay. It looks like there's only 3 options left:

Editing hosts files can obviously be done from the command-line, and that's the best way: sudo nano /etc/hosts I just researched available host-file-editor apps for MacOS and found the following 3 choices. I have NOT used them though, and am not sure if they're compatible with your OS version. You might ask the folks at Herd if they'd incorporate a local hostsfile editor into their Mac app.

driesvints commented 8 months ago

Closing this issue because it's inactive, already solved, old or not relevant anymore. Feel to open up a new issue if you're still experiencing this.