bug in Sniffing

[Alireza78na]

There is an issue with the Sniffing section that causes software like Spotify or some Google APIs to not route properly. For example, I have configured Xray to route geosite:Spotify traffic through Warp, but still a large portion of the APIs related to Spotify are not routing correctly.

I'm confident that the geosite file is complete and correct because enabling fake DNS resolves all issues, but unfortunately using fake DNS for all domains causes other problems.

Note that these issues have arisen after updating to the latest version and did not exist in previous versions.

Please look into this issue.

[Fangliding]

provid config, log and the last version that works properly

[Alireza78na]

log file: error.log

server Config:

{
  "api": {
    "services": [
      "HandlerService",
      "LoggerService",
      "StatsService"
    ],
    "tag": "api"
  },
  "inbounds": [
    {
      "listen": "127.0.0.1",
      "port": 62789,
      "protocol": "dokodemo-door",
      "settings": {
        "address": "127.0.0.1"
      },
      "tag": "api"
    }
  ],
  "log": {
    "access": "none",
    "dnsLog": false,
    "error": "none",
    "loglevel": "warning"
  },
  "outbounds": [
    {
      "protocol": "freedom",
      "settings": {},
      "tag": "direct"
    },
    {
      "protocol": "blackhole",
      "settings": {},
      "tag": "blocked"
    },
    {
      "tag": "warp",
      "protocol": "wireguard",
      "settings": {
        "mtu": 1420,
        "secretKey": "***********************************",
        "address": [
          "********"
        ],
        "workers": 2,
        "peers": [
          {
            "publicKey": "*******************************",
            "allowedIPs": [
              "0.0.0.0/0",
              "::/0"
            ],
            "endpoint": "************************",
            "keepAlive": 20
          }
        ],
        "kernelMode": false
      }
    },
    {
      "tag": "IPv4",
      "protocol": "freedom",
      "settings": {
        "domainStrategy": "UseIPv4"
      }
    }
  ],
  "policy": {
    "levels": {
      "0": {
        "statsUserDownlink": true,
        "statsUserUplink": true
      }
    },
    "system": {
      "statsInboundDownlink": true,
      "statsInboundUplink": true,
      "statsOutboundDownlink": true,
      "statsOutboundUplink": true
    }
  },
  "routing": {
    "domainStrategy": "IPIfNonMatch",
    "rules": [
      {
        "inboundTag": [
          "api"
        ],
        "outboundTag": "api",
        "type": "field"
      },
      {
        "ip": [
          "geoip:private"
        ],
        "outboundTag": "blocked",
        "type": "field"
      },
      {
        "outboundTag": "blocked",
        "protocol": [
          "bittorrent"
        ],
        "type": "field"
      },
      {
        "type": "field",
        "outboundTag": "blocked",
        "domain": [
          "geosite:speedtest"
        ]
      },
      {
        "type": "field",
        "domain": [
          "geosite:netflix"
        ],
        "outboundTag": "IPv4"
      },
      {
        "type": "field",
        "outboundTag": "warp",
        "domain": [
          "geosite:spotify"
        ]
      }
    ]
  },
  "stats": {},
  "dns": {
    "servers": [
      "94.140.14.14",
      "94.140.15.15",
      "2a10:50c0::ad1:ff",
      "2a10:50c0::ad2:ff"
    ],
    "queryStrategy": "UseIP",
    "tag": "dns_inbound"
  },
  "fakedns": null
}

last working version: 1.8.9

i should say that in the web browsing we usually dont have a problem for spotify etc. the problem appears when we use its Apps in Android , IOS , Windows etc. i use VMESS WS

[Fangliding]

I didn't see any valid inbound in your config

[Alireza78na]

I'm sorry, but what do you mean by config? I sent the server config. Do you mean the client config for connecting? Please explain a bit so I don't send the wrong files. Thank you.

[Fangliding]

Threre is only one inbound in your config and It's an api-in

[mmmray]

can you try blocking udp port 443? spotify uses quic, and I think the sniffer does not work on quic at all (see https://github.com/XTLS/Xray-core/issues/2616)

I am not sure why/how fakedns works, but I suppose since it connects dns lookups with udp/tcp traffic, it can do better sniffing.

[Alireza78na]

I think its because of using 3-xui panel. I don't use xray manually and I use MHSanaei panel. Maybe this is the reason why this is happening. In any case, I am ready to build a server with a similar configuration if necessary and provide it to you for further investigation.

[Alireza78na]

can you try blocking udp port 443? spotify uses quic, and I think the sniffer does not work on quic at all (see #2616)

I am not sure why/how fakedns works, but I suppose since it connects dns lookups with udp/tcp traffic, it can do better sniffing.

Thanks, i will try it and let you know the result here

[mmmray]

ok, i just saw that the config you posted doesn't have a sniffing config at all...

if it's not possible to extract the generated xray config from 3x-ui (including a complete inbound section) then it is impossible to have a productive conversation in the xray-core issue tracker. i suggest to investigate how to extract a full server config from 3x-ui, or ask on their support channels

[Alireza78na]

can you try blocking udp port 443? spotify uses quic, and I think the sniffer does not work on quic at all (see #2616)

I am not sure why/how fakedns works, but I suppose since it connects dns lookups with udp/tcp traffic, it can do better sniffing.

I tried this , but nothing solved for me. I blocked 443/udp in firewall for inbound and even outbound , but still problem not solved.

Also i will try to get full config from 3x-ui and send it here

[Alireza78na]

I should add this that many people also have my problem and We have repeatedly requested this matter from the panel creators, but a sufficient solution was not provided, and ultimately, we were referred to you. This issue is important because services like Spotify regularly block server IPs, and we cannot constantly change server IPs. The only solution is to use Warp When all traffic passes through Warp and we do not use sniffing, the issue is resolved. However, as you know, using it in this way significantly reduces the connection quality.

[mmmray]

We have repeatedly requested this matter from the panel creators, but a sufficient solution was not provided, and ultimately, we were referred to you.

the separation of responsibilities is important, and what exactly you are asking of them is important.

3x-ui is not responsible for supporting sniffing, but they are responsible IMO for providing a way to debug xray-core issue

if they don't have a simple way to get the full server config, xray-core developers will have to understand 3x-ui/marzban and debug it

[Fangliding]

If the problem only exists in 1.8.10, this may indeed be a bug To be honest, it's difficult to find valid information in the logs you provided It's best for you to write a config file, restart xray, and access Spotify once to reduce invalid information in the log Just a sentence, "I can't use Spotify", we can hardly help you

[us254]

~It could be due to the "IPIfNonMatch" domain strategy, where if the IP is already known (from cache or previous resolution), it takes precedence over the domain rule, See if changing the domain strategy to "AsIs" makes a difference in the routing decisions.~

[Alireza78na]

It could be due to the "IPIfNonMatch" domain strategy, where if the IP is already known (from cache or previous resolution), it takes precedence over the domain rule, See if changing the domain strategy to "AsIs" makes a difference in the routing decisions.

Thank you for your guidance, but this method didn't work. In fact, the settings were like this from the beginning.

[Alireza78na]

If the problem only exists in 1.8.10, this may indeed be a bug To be honest, it's difficult to find valid information in the logs you provided It's best for you to write a config file, restart xray, and access Spotify once to reduce invalid information in the log Just a sentence, "I can't use Spotify", we can hardly help you

Thank you for your patience and guidance. I created a separate server for testing and also prepared the debug log and config.json files.

I'm sending them here. If needed, I can give you server access to personally review them.

I should also mention that on this server, since the IP hasn't been blocked by Spotify yet, everything works correctly. However, I don't think it will affect the Sniffing investigation process.

If I mistakenly sent something wrong again, please let me know.

Log: error.log

Config: config.json

[RPRX]

Note that these issues have arisen after updating to the latest version and did not exist in previous versions.

v1.8.10 似乎没有修改 sniffer 相关代码

[Alireza78na]

Note that these issues have arisen after updating to the latest version and did not exist in previous versions.

v1.8.10 似乎没有修改 sniffer 相关代码

so maybe we have some issue in DNS, because my problem solved with this solution: #1019issue

please check it

[chise0713]

It could be due to the "IPIfNonMatch" domain strategy, where if the IP is already known (from cache or previous resolution), it takes precedence over the domain rule, See if changing the domain strategy to "AsIs" makes a difference in the routing decisions.

I've been wondering, is your understanding of the logic of IPIfNonMatch correct?

Because the document and the source code seems shows that IPIfNonMatch is not work as what you tought. There is no "from cache or previous resolution", it is always use domain to match first, if it doesn't match any of domain rules, it will resovle domain to IP address then match rules again.

[us254]

It could be due to the "IPIfNonMatch" domain strategy, where if the IP is already known (from cache or previous resolution), it takes precedence over the domain rule, See if changing the domain strategy to "AsIs" makes a difference in the routing decisions.

I've been wondering, is your understanding of the logic of IPIfNonMatch correct?

Because the document and the source code seems shows that IPIfNonMatch is not work as what you tought. There is no "from cache or previous resolution", it is always use domain to match first, if it doesn't match any of domain rules, it will resovle domain to IP address then match rules again.

you are correct. Thank you for bringing this to my attention

[RPRX]

Note that these issues have arisen after updating to the latest version and did not exist in previous versions.

v1.8.10 似乎没有修改 sniffer 相关代码

so maybe we have some issue in DNS, because my problem solved with this solution: #1019issue

please check it

v1.8.10 似乎也没有修改 dns 相关代码，请逐个测试近期的 commits，看一下哪个 commit 出现了你的问题

@Fangliding 等会儿在 issue 模板中加一下吧，如果新版有 bug 需先测出是哪个 commit 的问题再开 issue

[pulsarice]

I did some testing on spotify app on android. Spotify has blocked my server's IP address, so I have to route spotify.com to WARP (Quic protocol is disabled in both cases.)

First without fakeDNS (Spotify doesn't work)

accepted tcp:graph.facebook.com:443 [inbound -> warp] 
accepted tcp:graph.facebook.com:443 [inbound -> warp] 
accepted tcp:clienttoken.spotify.com:443 [inbound -> warp] 
accepted tcp:apresolve.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-dealer.spotify.com:443 [inbound -> warp] 
accepted tcp:relaycdn.anchor.fm:443 [inbound >> direct] 
accepted tcp:34.158.0.131:4070 [inbound >> direct] 
accepted tcp:spclient.wg.spotify.com:443 [inbound -> warp] 
accepted tcp:spclient.wg.spotify.com:443 [inbound -> warp] 
accepted tcp:spclient.wg.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-dealer.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 

Now with fakeDNS enabled (Spotify works)

accepted tcp:graph.facebook.com:443 [inbound -> warp] 
accepted tcp:graph.facebook.com:443 [inbound -> warp] 
accepted tcp:spclient.wg.spotify.com:443 [inbound -> warp] 
accepted tcp:spclient.wg.spotify.com:443 [inbound -> warp] 
accepted tcp:apresolve.spotify.com:443 [inbound -> warp] 
accepted tcp:spclient.wg.spotify.com:443 [inbound -> warp] 
accepted tcp:apresolve.spotify.com:443 [inbound -> warp] 
accepted tcp:relaycdn.anchor.fm:443 [inbound >> direct] 
accepted tcp:login5.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-dealer.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:gew4-spclient.spotify.com:443 [inbound -> warp] 
accepted tcp:ap-gew4.spotify.com:4070 [inbound -> warp] 
accepted tcp:gew4-dealer.spotify.com:443 [inbound -> warp] 
accepted tcp:image-cdn-ak.spotifycdn.com:443 [inbound >> direct] 

The difference is that tcp:34.158.0.131:4070 is now tcp:ap-gew4.spotify.com:4070 It seems that sniffing didn't catch that. Is it because of a possible non-standard port/protocol used by spotify app?

[mmmray]

I just took a look with wireshark. Yes, from what I can tell port 4070 is an entirely proprietary protocol, and xray simply does not have support for sniffing a domain name from it.

i don't think there's a bug, only perhaps the absence of a feature. Correct me if I'm wrong, but xray's "geosite" support requires domain name sniffing to apply, and figuring out the domain name from a tcp connection is either a per-protocol thing to support (http, tls, ...), or something to bypass entirely with the big hammer of fakedns.

I think you can fix it with fakedns as you did here or write a rule that simply routes the entire IP 34.158.0.131 to warp. The IP belongs to Google, but since it accepts a custom protocol on a custom port, I find it very unlikely that spotify shares it with any other google cloud customer.

[pulsarice]

write a rule that simply routes the entire IP 34.158.0.131 to warp

Or blindly route all traffic with destination port of 4070 to warp :)) Is any other known service using that port number? Thank you for investigating the matter.

[yuhan6665]

Thanks for the great investigation @pulsarice @mmmray

[Fangliding]

Wait, why these did not happen in 1.8.9

[mmmray]

Is any other known service using that port number?

No idea, but it seems much more likely, statistically. I suggest to add the IP to geosite repo :man_shrugging:

Wait, why these did not happen in 1.8.9

No idea. Did anybody actually try downgrading? I wonder if spotify simply updated around the time xray 1.8.10 was released.

[Fangliding]

@mmmray GEOSITE only contains domain, no IP and maybe you can try to use 1.8.9 and see how this “protocol 4070” be routed

[yuhan6665]

Wait, why these did not happen in 1.8.9

There is no change on our side, most likely user don't know/bother to test each version (I think I cannot guarantee to use binary search to find the problematic version for an issue as well) This is the essence of any large product support, you are diving in mud water to find thing ;) On the bright side, it make me twice as happy to see people dig the gold (root cause) from mud

[mmmray]

GEOSITE only contains domain, no IP

I wonder, can it be changed? I think supporting IPs in geosite file format would help a lot with proprietary protocols.

I do not have a working xray routing setup at all at the moment, I only looked at xray docs and spotify traffic, if nobody else does it i can do it next week ~or i forget about it entirely~

[yuhan6665]

There is a separate project for IP https://github.com/Loyalsoldier/geoip Although I understand the goal of this project is to sort based on geolocation. It might not be realistic to have more categories

[pulsarice]

Wait, why these did not happen in 1.8.9

It started happening to me many months ago, after a specific version update of spotify app. I actually downgraded my spotify app back then and can confirm that older version worked.

[Fangliding]

GEOSITE only contains domain, no IP

I wonder, can it be changed? I think supporting IPs in geosite file format would help a lot with proprietary protocols.

I do not have a working xray routing setup at all at the moment, I only looked at xray docs and spotify traffic, if nobody else does it i can do it next week ~or i forget about it entirely~

Some special IP ranges may be contained by some third-party geoip However, the IP is different from the domain, and due to the existence of services such as CDN and balancers, services on one IP are likely to be frequently replaced

[Fangliding]

Wait, why these did not happen in 1.8.9

It started happening to me many month ago, after a specific version update of spotify app. I actually downgraded my spotify app back then and can confirm that older version worked.

So nobody sure if it's working correct on 1.8.9............. Damn I need add this to template.... Perhaps one day it will become as complex as aviation safety regulations. Ordinary people think they are outrageous, but there are even more outrageous things behind them

[pulsarice]

So you're not sure if it's working correct on 1.8.9

v1.8.9 wasn't even conceived when this started happening to me.

[Fangliding]

So you're not sure if it's working correct on 1.8.9

v1.8.9 wasn't even conceived when this started happening to me.

I mean the author of this issue, I read the wrong person Edited

[Kiya6955]

I just took a look with wireshark. Yes, from what I can tell port 4070 is an entirely proprietary protocol, and xray simply does not have support for sniffing a domain name from it.

i don't think there's a bug, only perhaps the absence of a feature. Correct me if I'm wrong, but xray's "geosite" support requires domain name sniffing to apply, and figuring out the domain name from a tcp connection is either a per-protocol thing to support (http, tls, ...), or something to bypass entirely with the big hammer of fakedns.

I think you can fix it with fakedns as you did here or write a rule that simply routes the entire IP 34.158.0.131 to warp. The IP belongs to Google, but since it accepts a custom protocol on a custom port, I find it very unlikely that spotify shares it with any other google cloud customer.

This ip was working for me till past week. This week i had problem again and i found new ip address 35.186.224.28. Dst ports was different so i decided to route whole traffic of this ip behind Warp. Now im using

      {
        "outboundTag": "warp",
        "ip": [
          "35.186.224.28"
        ],
        "type": "field"
      },
      {
        "outboundTag": "warp",
        "domain": [
          "geosite:spotify"
        ],
        "type": "field"
      }

[pulsarice]

This ip was working for me till past week. This week i had problem again and i found new ip address 35.186.224.28. Dst ports was different so i decided to route whole traffic of this ip behind Warp. Now im using

      {
        "outboundTag": "warp",
        "ip": [
          "35.186.224.28"
        ],
        "type": "field"
      },
      {
        "outboundTag": "warp",
        "domain": [
          "geosite:spotify"
        ],
        "type": "field"
      }

I simply routed port 4070 and spotify.com domain to warp and haven't had any problems since:

  {
    "type": "field",
    "domain": [
      "domain:spotify.com"
    ],
    "outboundTag": "warp"
  },
  {
    "type": "field",
    "port": "4070",
    "outboundTag": "warp"
  }