MTR is showing incorrect times on last hop when doing TCP

[Xymox1]

The last hop shows the wrong latency. Tested on 3 different machines on 3 different OS versions.

See this for discussion. https://www.dslreports.com/forum/r31122204-SB6190-Puma6-TCP-UDP-Network-Latency-Issue-Discussion~start=4591

[rewolff]

MTR is unlikely to pull measurements out of its thumb. It's a simple computer program, not an AI programmed to lie to you.

When I use MTR to try to trace to MY webserver, using TCP, I get "expected results" (< 1ms to the ADSL modem, 30ms to the next hop, <1ms to the server, 100km down the road... i.e. NOT the 40ms of delay that you're seeing for that last hop to your destination host.

When I prod MY server, the server does not initiate the TCPIP connection, it responds to the packet with an ICMP message. I'm guessing that this is RFC compliant.

I tested YOUR host, but I could not contact that host's port 53. I'm guessing it is your nameserver and that it's configured for helping you and not me. Your host is firewalled from me.

There could be two things going on. First the host might be behind a load-distributing machine, that causes the delays.... in some cases. Second the target host might take 40ms to process the slightly off-spec connection request. If it sets up a new TCP connection and responds with SYN+ACK, it is conceivable that this takes some extra processing time.

Because MTR and similar applications work by sending "funny" packets, there are lots of variations on exactly how funny those packets are. It is possible that mtr made a slightly different packet than that other trace program of yours, causing a slightly different measurement.

As it stands, it is a "cannot reproduce": My webserver shows less than 0.5 ms delay from the next-to-last hop. Your example host (cdns1.cox.net, 68.105.28.11) does not respond to my probes.

If you could tcpdump or wireshark a session we could analyse the stuff on the wire.

[Xymox1]

I had the same results targeting google on port 80. A 5 times increase in the last hop VS Pingplotter running at the same time. The rest of the hops are the same in both programs. Only the last hop displays this weird behavior. It does this on every target I have aimed it against. I also targeted pingplotter.com. I also of course tested other sites and had the same result. I suggest you download pingplotter free trial. Also a ICMP ping shows a very different, and correct , latency.

[Xymox1]

Ive used Pingplotter for years against a large number of targets doing TCP/UDP. So im very used to doing this and have a lot of experence with the numbers. MTR is reporting incorrect numbers for the last hop. Its correct for all hops up to the last one. But the last one is wrong.

[yvs2014]

It looks a bit like this https://github.com/traviscross/mtr/issues/95

[rewolff]

I analized this yesterday and indeed the destination host replies with "expected" response times. Then some more tcp handshaking happens, and I was planning on staring at the TCPDUMP some more to figure out which one was counted as the reply.

@yvs2014 , it looks like you submitted a patch. Did I pull your patch, or did I forget? It could very well be that I'm running an older version at home when I looked at this yesterday.

[yvs2014]

Sorry, but what version is it about? (I'm not too familiar with last versions and just cannot find any TCP processing in the current code)

[matt-kimball]

For TCP probes, mtr measures the time for intermediate hops as the delay between sending the initial SYN packet to receiving an ICMP TTL Expired reply. For the destination, the measures the time for the TCP connection to be considered full established by the host operating system. (i.e. for the TCP handshake to complete.)

It might be interesting to see if different timing is observed if instead of waiting for the host operating system to complete the TCP handshake, instead the time to the receiving the destination's TCP SYN-ACK via a raw socket. If the timings are significantly different, then the local TCP stack on the host doing measurements would be to blame for the difference in timing.

For what it's worth, the code which checks for TCP connection completion is in receive_replies_from_probe_socket in probe_unix.c -- the select call in that function is the check for completion, and if the socket connection has completed, the time is measured in the subsequent call to receive_probe.

I'm concerned that moving to measuring the time when the SYN-ACK packet is received via a raw socket will lead to "interesting" portability issues, but it is probably a better approach because it will give more precise timing.

[matt-kimball]

To further explain the portability issues -- I am very concerned that receiving a raw SYN-ACK packet (as application level code) for a TCPv6 connection on MacOS may be impossible, and possibly on other BSD operating systems as well.

[LionNatsu]

lion@Lion-Laptop [ ~ ] $ mtr 10.0.0.1 --tcp --port 80 --report -c 2
Start: Wed May 31 08:00:07 2017
HOST: Lion-Laptop                 Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- Lion-Router0               0.0%     2  1000. 601.1 201.5 1000. 565.1
lion@Lion-Laptop [ ~ ] $ mtr --version
mtr 0.87
lion@Lion-Laptop [ ~ ] $ 

lion@Lion-Laptop [ ~ ] $ tshark -Y ip.addr==10.0.0.1
Capturing on 'enp5s0'
    1 0.000000000 00:00:07.712515671     10.0.2.1 → 10.0.0.1     TCP 66 33461 → 80 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128
    2 0.000379166 00:00:07.712894837     10.0.0.1 → 10.0.2.1     TCP 66 80 → 33461 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1 WS=2
    3 0.000418955 00:00:07.712934626     10.0.2.1 → 10.0.0.1     TCP 54 33461 → 80 [ACK] Seq=1 Ack=1 Win=29312 Len=0
    4 0.101214897 00:00:07.813730568     10.0.2.1 → 10.0.0.1     TCP 66 46471 → 80 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128
    5 0.101661568 00:00:07.814177239     10.0.0.1 → 10.0.2.1     TCP 66 80 → 46471 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1 WS=2
    6 0.101713812 00:00:07.814229483     10.0.2.1 → 10.0.0.1     TCP 54 46471 → 80 [ACK] Seq=1 Ack=1 Win=29312 Len=0
    7 0.201717813 00:00:07.914233484     10.0.2.1 → 10.0.0.1     TCP 55 [TCP segment of a reassembled PDU]
    8 0.201759867 00:00:07.914275538     10.0.2.1 → 10.0.0.1     TCP 54 33461 → 80 [FIN, ACK] Seq=2 Ack=1 Win=29312 Len=0
    9 0.202134665 00:00:07.914650336     10.0.0.1 → 10.0.2.1     TCP 60 80 → 33461 [ACK] Seq=1 Ack=2 Win=5840 Len=0
   10 0.202486963 00:00:07.915002634     10.0.2.1 → 10.0.0.1     TCP 66 60923 → 80 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128
   11 0.202921444 00:00:07.915437115     10.0.0.1 → 10.0.2.1     TCP 66 80 → 60923 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1 WS=2
   12 0.202971648 00:00:07.915487319     10.0.2.1 → 10.0.0.1     TCP 54 60923 → 80 [ACK] Seq=1 Ack=1 Win=29312 Len=0
   13 0.205581698 00:00:07.918097369     10.0.0.1 → 10.0.2.1     TCP 404 [TCP segment of a reassembled PDU]
   14 0.205637536 00:00:07.918153207     10.0.2.1 → 10.0.0.1     TCP 54 33461 → 80 [RST] Seq=3 Win=0 Len=0
   15 0.205718176 00:00:07.918233847     10.0.0.1 → 10.0.2.1     HTTP 60 HTTP/1.0 400 Invalid Request  (text/html)
   16 0.205735544 00:00:07.918251215     10.0.2.1 → 10.0.0.1     TCP 54 33461 → 80 [RST] Seq=3 Win=0 Len=0
   17 0.303060720 00:00:08.015576391     10.0.2.1 → 10.0.0.1     TCP 55 [TCP segment of a reassembled PDU]
   18 0.303102889 00:00:08.015618560     10.0.2.1 → 10.0.0.1     TCP 54 46471 → 80 [FIN, ACK] Seq=2 Ack=1 Win=29312 Len=0
   19 0.303476375 00:00:08.015992046     10.0.0.1 → 10.0.2.1     TCP 60 80 → 46471 [ACK] Seq=1 Ack=2 Win=5840 Len=0
   20 0.303782834 00:00:08.016298505     10.0.2.1 → 10.0.0.1     TCP 66 56667 → 80 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128
   21 0.304146445 00:00:08.016662116     10.0.0.1 → 10.0.2.1     TCP 66 80 → 56667 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1 WS=2
   22 0.304180043 00:00:08.016695714     10.0.2.1 → 10.0.0.1     TCP 54 56667 → 80 [ACK] Seq=1 Ack=1 Win=29312 Len=0
   23 0.305177459 00:00:08.017693130     10.0.0.1 → 10.0.2.1     TCP 404 [TCP segment of a reassembled PDU]
   24 0.305233252 00:00:08.017748923     10.0.2.1 → 10.0.0.1     TCP 54 46471 → 80 [RST] Seq=3 Win=0 Len=0
   25 0.305313747 00:00:08.017829418     10.0.0.1 → 10.0.2.1     HTTP 60 HTTP/1.0 400 Invalid Request  (text/html)
   26 0.305329726 00:00:08.017845397     10.0.2.1 → 10.0.0.1     TCP 54 46471 → 80 [RST] Seq=3 Win=0 Len=0
   27 0.637981922 00:00:08.350497593     10.0.2.1 → 10.0.0.1     TCP 55 [TCP segment of a reassembled PDU]
   28 0.638009707 00:00:08.350525378     10.0.2.1 → 10.0.0.1     TCP 54 60923 → 80 [FIN, ACK] Seq=2 Ack=1 Win=29312 Len=0
   29 0.638402615 00:00:08.350918286     10.0.0.1 → 10.0.2.1     TCP 60 80 → 60923 [ACK] Seq=1 Ack=2 Win=5840 Len=0
   30 0.639463183 00:00:08.351978854     10.0.0.1 → 10.0.2.1     TCP 404 [TCP segment of a reassembled PDU]
   31 0.639519497 00:00:08.352035168     10.0.2.1 → 10.0.0.1     TCP 54 60923 → 80 [RST] Seq=3 Win=0 Len=0
   32 0.639578453 00:00:08.352094124     10.0.0.1 → 10.0.2.1     HTTP 60 HTTP/1.0 400 Invalid Request  (text/html)
   33 0.639593480 00:00:08.352109151     10.0.2.1 → 10.0.0.1     TCP 54 60923 → 80 [RST] Seq=3 Win=0 Len=0
   36 1.304785924 00:00:09.017301595     10.0.2.1 → 10.0.0.1     TCP 55 [TCP segment of a reassembled PDU]
   37 1.304813432 00:00:09.017329103     10.0.2.1 → 10.0.0.1     TCP 54 56667 → 80 [FIN, ACK] Seq=2 Ack=1 Win=29312 Len=0
   38 1.305192961 00:00:09.017708632     10.0.0.1 → 10.0.2.1     TCP 60 80 → 56667 [ACK] Seq=1 Ack=2 Win=5840 Len=0
   39 1.306253142 00:00:09.018768813     10.0.0.1 → 10.0.2.1     TCP 404 [TCP segment of a reassembled PDU]
   40 1.306300680 00:00:09.018816351     10.0.2.1 → 10.0.0.1     TCP 54 56667 → 80 [RST] Seq=3 Win=0 Len=0
   41 1.306395237 00:00:09.018910908     10.0.0.1 → 10.0.2.1     HTTP 60 HTTP/1.0 400 Invalid Request  (text/html)
   42 1.306408730 00:00:09.018924401     10.0.2.1 → 10.0.0.1     TCP 54 56667 → 80 [RST] Seq=3 Win=0 Len=0
   52 6.314530553 00:00:14.027046224     10.0.2.1 → 10.0.0.1     DNS 81 Standard query 0xd745 PTR 1.0.0.10.in-addr.arpa
   53 6.315653420 00:00:14.028169091     10.0.0.1 → 10.0.2.1     DNS 107 Standard query response 0xd745 PTR 1.0.0.10.in-addr.arpa PTR Lion-Router0
^C42 packets captured
lion@Lion-Laptop [ ~ ] $ 

No ICMP message was received. 10.0.0.1 is a router with Tomato operating system.

[matt-kimball]

Sure. No ICMP messages were received because your destination was one hop away from your source. If your destination host was more than one hop away, you would see ICMP TTL expired packets from some intermediate hosts.

[LionNatsu]

Another test for two hop (on my server). h -> x -> y

Start: Wed May 31 09:41:46 2017
HOST: hhh.hhh.111.193             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- xxx.xxx.110.3              0.0%     1    3.3   3.3   3.3   3.3   0.0
  2.|-- yyy.yyy.41.129             0.0%     1  200.8 200.8 200.8 200.8   0.0

   45 3.507340336 hhh.hhh.111.193 → yyy.yyy.41.129 TCP 74 42743 → 80 [SYN] Seq=0 Win=42340 Len=0 MSS=1460 SACK_PERM=1 TSval=114132510 TSecr=0 WS=2048
   46 3.510530615 xxx.xxx.110.3 → hhh.hhh.111.193 ICMP 70 Time-to-live exceeded (Time to live exceeded in transit)
   47 3.607901813 hhh.hhh.111.193 → yyy.yyy.41.129 TCP 74 30789 → 80 [SYN] Seq=0 Win=42340 Len=0 MSS=1460 SACK_PERM=1 TSval=3826855996 TSecr=0 WS=2048
   48 3.608881670 yyy.yyy.41.129 → hhh.hhh.111.193 TCP 54 80 → 30789 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
   49 3.708469622 hhh.hhh.111.193 → yyy.yyy.41.129 TCP 74 13779 → 80 [SYN] Seq=0 Win=42340 Len=0 MSS=1460 SACK_PERM=1 TSval=967085748 TSecr=0 WS=2048
   50 3.709501009 yyy.yyy.41.129 → hhh.hhh.111.193 TCP 54 80 → 13779 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
   52 3.809050639 hhh.hhh.111.193 → yyy.yyy.41.129 TCP 74 16831 → 80 [SYN] Seq=0 Win=42340 Len=0 MSS=1460 SACK_PERM=1 TSval=3528720764 TSecr=0 WS=2048
   53 3.810051019 yyy.yyy.41.129 → hhh.hhh.111.193 TCP 54 80 → 16831 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

It was less than 5ms in ICMP mode but it is ~200ms in TCP.

[LionNatsu]

Same version of mtr, to a port without service.

[LionNatsu]

The address might be confused... Edited.

[LionNatsu]

There are 3 times than one shot on destination. Timeline: RTT, 100ms, RTT, 100ms, RTT

[matt-kimball]

Yes -- that looks like a good demonstration that the problem is the way that mtr is measuring the time for completed TCP sockets.

[LionNatsu]

Not quite. connect() returns after received SYN+ACK or RST+ACK, which only takes few millisecond (0.8ms, 1.1ms, 1.0ms in the server example). The only reason can "boost" the latency up to hundreds of milliseconds is the intervals by mtr itself, and the method to compute the duration.

What is the 100ms intervals? Why does it do connect (SYN) 3 times even when -c 1? What is the 1000ms (see the router example) in the Round 2?

[matt-kimball]

For what it's worth, I am able to reproduce this issue with mtr v0.87, but it appears to be fixed when using the latest code from github.

The code current in github -- which hasn't yet made it into a release -- has had the packet handling nearly entirely rewritten, so it's not surprising that it behaves differently. Perhaps that rewrite has unintentionally fixed this issue.

Also, the fact that the latest code appears to fix the issue invalidates my theory that it was the local TCP stack introducing a delay. I was wrong about that.

@LionNatsu, can you try building the latest from github and see if the issue is fixed?

[LionNatsu]

Sure, I will rebuild it this night. ("twelve hours latel.")

[LionNatsu]

Brilliant, I rebuilt just now, at tip of branch master. This bug has been fixed.

root@Lion-Laptop [ test@master ] ! mtr github.com --tcp --port 80 --report -c 2
Start: 2017-05-31T12:19:25+0800
HOST: Lion-Laptop                 Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- Lion-Router0               0.0%     2    0.4   0.4   0.4   0.4   0.0
...
 12.|-- GITHUB-INC.ear3.Seattle1.  0.0%     2  244.6 247.9 244.6 251.1   4.6
 13.|-- 207.88.12.158.ptr.us.xo.n  0.0%     2  201.4 193.6 185.7 201.4  11.1
 14.|-- ae0d0.cir1.sanjose2-ca.us  0.0%     2  207.0 206.3 205.5 207.0   1.0
 15.|-- 192.30.255.112             0.0%     2  238.7 235.2 231.8 238.7   4.8
root@Lion-Laptop [ test@master ] # 

Router test:

root@Lion-Laptop [ test@master ] # mtr 10.0.0.1 --tcp --port 80 --report -c 1
Start: 2017-05-31T12:23:28+0800
HOST: Lion-Laptop                 Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- Lion-Router0               0.0%     1    0.7   0.7   0.7   0.7   0.0

lion@Lion-Laptop [ ~ ] $ tshark -Y ip.addr==10.0.0.1
Capturing on 'enp5s0'
    1 0.000000000 04:23:29.067367287     10.0.2.1 → 10.0.0.1     TCP 66 33000 → 80 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128
    2 0.000470890 04:23:29.067838177     10.0.0.1 → 10.0.2.1     TCP 66 80 → 33000 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1 WS=2
    3 0.000536841 04:23:29.067904128     10.0.2.1 → 10.0.0.1     TCP 54 33000 → 80 [ACK] Seq=1 Ack=1 Win=29312 Len=0
    4 0.000656819 04:23:29.068024106     10.0.2.1 → 10.0.0.1     TCP 54 33000 → 80 [FIN, ACK] Seq=1 Ack=1 Win=29312 Len=0
    5 0.003409933 04:23:29.070777220     10.0.0.1 → 10.0.2.1     TCP 404 [TCP segment of a reassembled PDU]
    6 0.003468561 04:23:29.070835848     10.0.2.1 → 10.0.0.1     TCP 54 33000 → 80 [RST] Seq=2 Win=0 Len=0
    7 0.100154060 04:23:29.167521347     10.0.2.1 → 10.0.0.1     TCP 66 33001 → 80 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128
    8 0.100566516 04:23:29.167933803     10.0.0.1 → 10.0.2.1     TCP 66 80 → 33001 [SYN, ACK] Seq=0 Ack=1 Win=5840 Len=0 MSS=1460 SACK_PERM=1 WS=2
    9 0.100619908 04:23:29.167987195     10.0.2.1 → 10.0.0.1     TCP 54 33001 → 80 [ACK] Seq=1 Ack=1 Win=29312 Len=0
   10 0.100703284 04:23:29.168070571     10.0.2.1 → 10.0.0.1     TCP 54 33001 → 80 [FIN, ACK] Seq=1 Ack=1 Win=29312 Len=0
   11 0.103375848 04:23:29.170743135     10.0.0.1 → 10.0.2.1     TCP 404 [TCP segment of a reassembled PDU]
   12 0.103404735 04:23:29.170772022     10.0.2.1 → 10.0.0.1     TCP 54 33001 → 80 [RST] Seq=2 Win=0 Len=0
^C12 packets captured

version: mtr 0.87.227-9d2f

[LionNatsu]

In the short term, I believe the project needs a release to fix this issue which overestimate the latency too much... (it's about a year and 227 commits to master since latest release)

[Xymox1]

Well awesome. good job :) Yes a release sounds due :)

[rewolff]

I think we should blame yvs2014 for the fix, and I accepted his patch.

My "make a release" script is still "under supervison".... as in "last time it didn't work as planned". And I forgot what went wrong, so I'm a bit afraid to just "run it and see what happens". In the past things have gotten confused with patches not correctly moving between releases and release tarballs not being complete... I should gather some courage sometime....

[LionNatsu]

@rewolff Cool. Why not duplicate (fork) the repo and run the script on the new repo (playground) and see what will happens? It might be useful to debug your script, no need to gather courage ;)

[rewolff]

OK. Guys. Sami Kerola broke the build on my systems.

m4_esyscmd is "possibly undefined", which a program that refuses to identify itself finds considers an error. Personally I would really prefer it if A) programs would identify themselves, B) they are SURE that something is undefined before considering something an error.

The odd thing is: on those same machines a clean git clone of the master on github does work.

FOUND IT!!!!

the "bootstrap.sh" fails for releases, but works for intermediate versions.

So... for your information... I have 0.88 on my machine, but it fails to build.... I am hesitant to push it to github in that state.

[LionNatsu]

@rewolff I found that if you clone with argument depth=1, ./bootstrap.sh && ./configure --version will get mtr UNKNOWN because there is no tag information. But cloning the whole repo works: mtr configure 0.87.227-9d2f.

Did you mean this?

[rewolff]

My mtr is now "version 0.88".... but it fails to compile (in bootstrap.sh) because MAYBE m4_esyscmd is undefined. But I can't find any definition for that, nor does it NEED to be defined in MTR because on 0.87.227-9d2f you do not seem to need a definition for it. The error message tells me to look into the documentation for autoconf. But the manual does not once mention m4_esyscmd.

[matt-kimball]

Maybe it is time to consider releasing version 1.0 soon?

This December will be 20 years since the first version of mtr. I uploaded the source code to sunsite.unc.edu and posted an announcement on comp.os.linux.announce in early December of 1997.

[rewolff]

I don't like playing political games with version numbers. But MTR has been reasonably stable the last few years and the impression that 0.xx gives is not that of something that has matured and is mostly stable.

We could pull a Donald Knuth joke to start to converge to some irrational number. sqrt(2) or something (pi and e are taken IIRC).

[matt-kimball]

You are free to choose a version number you like, but 0.xx does give the impression that things are incomplete or unstable. I think neither of those are true.

[rewolff]

About 12 releases from now I'm FORCED to think about that... :-) (I'm not going from 0.99 to 0.100, although my script would, in its current form, actually try do that....)

[Xymox1]

GO FOR 1.00 !!! A moment in MTR history !! I would be proud I triggered events that led to 1.0 ! Maybe you need a RC first ? 1.00RC hahaha..

[yvs2014]

It's about possibly undefined m4_esyscmd macro. As far as I can see, it's from AC_INIT in configure.ac file and might be temporarily escaped by something like this

--- a/configure.ac
+++ b/configure.ac
@@ -1,6 +1,7 @@
 AC_PREREQ([2.59])
+m4_define([my_version], [m4_ifdef([m4_esyscmd], [m4_esyscmd([build-aux/git-version-gen .tarball-version])], [0.88])])
 AC_INIT([mtr],
-  [m4_esyscmd([build-aux/git-version-gen .tarball-version])],
+  [my_version],
   [R.E.Wolff@BitWizard.nl], [],
   [http://www.BitWizard.nl/mtr/])
 AC_CONFIG_SRCDIR([ui/mtr.c])

But why that esyscmd is undefined in your case, it's unclear.

[LionNatsu]

Is the code the same (0.87.227) as master at GitHub? I want to know how to reproduce your compile error... Just give it a git tag called v0.88?

[LionNatsu]

@yvs2014 @rewolff FYI, I found something may be relevant.

Bug 1086985 - Nightly build failure: configure.ac:10: error: possibly undefined macro: m4_esyscmd

[rewolff]

You were right! Brilliant debugging: it was my make-a-release script that messed up, but you don't have my script. it was my script that changed a file on my machine, but you haven't seen those changes.

[LionNatsu]

@rewolff Strange. Tarball at https://github.com/traviscross/mtr/releases/tag/v0.88 fails as what you've said,

lion@Lion-Laptop [ mtr-0.88 ] $ ./bootstrap.sh 
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/\${ <-- HERE ([^ \t=:+{}]+)}/ at /bin/automake line 3936.
configure.ac:8: installing 'build-aux/compile'
configure.ac:20: installing 'build-aux/config.guess'
configure.ac:20: installing 'build-aux/config.sub'
configure.ac:9: installing 'build-aux/install-sh'
configure.ac:9: installing 'build-aux/missing'
Makefile.am: installing 'build-aux/depcomp'
parallel-tests: installing 'build-aux/test-driver'
configure.ac:3: error: possibly undefined macro: m4_esyscmd
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.

[LionNatsu]

OMG @rewolff @matt-kimball You must do something now. The tag v0.88 is on the wrong branch!

*   b8ffc0d (HEAD -> master, origin/master, origin/HEAD) Merge pull request #197 from fmazu/json_asn
|\  
| * 07c0ece Add AS number to json output
|/  
* 7496654 The release script bumped the version number
| * 80b4238 (tag: v0.88) The release script bumped the version number
|/  
*   9d2fa2d Merge pull request #195 from dajohi/select
|\  
| * 40e1b2b include <sys/select.h> for fd_set
|/  
*   e6d0a7e Merge pull request #192 from jwilk/spelling
...

[rewolff]

In the old days, my make-a-release script could be aborted if I saw some patch not included or too much included. That was not yet handled correctly. So tags v0.88 and v0.90 now point somewhere where I don't want them to point. I don't think it's good to "change history", so I've just re-tested my script. Getting better every time. :-)