search

internet-equity / netrics

The network measurements scheduling framework
7 stars 3 forks source link

Runtime error: "host network interface down" #38

Open marcwitasee opened 1 year ago

marcwitasee commented 1 year ago

I ran netrics debug run ping and received the following output:

Name: ping
Command: /Users/marcwitasee/test_netrics/venv/bin/netrics-ping

Status: Error (Exit code 71)

Result: -

Logged (standard error):

  <5> {"dest": "localhost", "status": "Error", "msg": "host network interface down"}

Running netrics.d --foreground also produces the error:

ℹ️  time="2023-04-05 12:57:39.960" level="info" event="3efEnxQRrqiA" session="3efEnxQRumoV" execution_count=0 scheduled_next=2023-04-05T12:58:00
☠️  time="2023-04-05 12:58:00.229" level="critical" event="3efEo7y1Xe3o" exc_line="base.py:27" exc_frame="verify" msg="fatal exception of type ProcessExecutionError"
netrics.d: fatal: ProcessExecutionError: Unexpected exit code: 71
Command line: | /Users/marcwitasee/test_netrics/venv/bin/netrics-ping
Stderr:       | <5> {"dest": "localhost", "status": "Error", "msg": "host network interface down"}
marcwitasee commented 1 year ago

Here is the traceback error:

ℹ️  time="2023-04-05 14:09:29.762" level="info" event="3efGlVv4iGOI" session="3efGlVv4lcU9" sched="tiered-tenancy" task="hops-scamper" msg="skipped: suppressed by if/unless condition"
ℹ️  time="2023-04-05 14:09:29.764" level="info" event="3efGlVvhSO3w" session="3efGlVv4lcU9" sched="tiered-tenancy" task="lml-scamper" msg="skipped: suppressed by if/unless condition"
Traceback (most recent call last):
  File "/Users/marcwitasee/test_netrics/venv/bin/netrics.d", line 8, in <module>
    sys.exit(daemon())
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/netrics/__main__.py", line 25, in entrypoint
    hook(
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/cli/root.py", line 31, in entrypoint
    argcmdr.main(root, extend_parser=partial(extend_parser, **settings))
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/argcmdr.py", line 76, in main
    command.call()
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/argcmdr.py", line 377, in call
    return self.delegate('__call__', *additional)
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/argcmdr.py", line 397, in delegate
    return target_callable(*call_args[:param_count])
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/cli/command/control.py", line 170, in __call__
    self.daemon(args.scheduler, background=background)
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/cli/command/control.py", line 189, in daemon
    run_info = self.serve(scheduler)
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/cli/command/control.py", line 204, in serve
    for completed_task in completed_tasks:
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/util/iteration.py", line 52, in __next__
    return next(self.iterator)
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/sched/base/scheduler.py", line 59, in __call__
    count = yield from self.exec_tasks(reset=reset)
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/sched/tiered_tenancy.py", line 85, in exec_tasks
    count_ready = yield from pool.iter_ready(refill=queue.tenancy_tasks(pool.size))
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/sched/base/pool.py", line 98, in iter_ready
    if task.ready():
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/fate/sched/base/scheduled_task.py", line 97, in poll
    if ready := self.__future__.poll():
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/plumbum/commands/modifiers.py", line 33, in poll
    self.wait()
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/plumbum/commands/modifiers.py", line 43, in wait
    self._returncode, self._stdout, self._stderr = run_proc(
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/plumbum/commands/processes.py", line 304, in run_proc
    return _check_process(proc, retcode, timeout, stdout, stderr)
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/plumbum/commands/processes.py", line 17, in _check_process
    proc.verify(retcode, timeout, stdout, stderr)
  File "/Users/marcwitasee/test_netrics/venv/lib/python3.9/site-packages/plumbum/machines/base.py", line 27, in verify
    raise ProcessExecutionError(
plumbum.commands.processes.ProcessExecutionError: Unexpected exit code: 71
Command line: | /Users/marcwitasee/test_netrics/venv/bin/netrics-dns-latency
Stderr:       | <5> {"dest": "localhost", "status": "Error", "msg": "host network interface down"}
jesteria commented 1 year ago

All right:

So, (outside of Netrics), you can't just do ping localhost? What are the contents of your /etc/hosts? (Are you running an antivirus application or a special firewall?)

marcwitasee commented 1 year ago
marcwitasee commented 1 year ago

Okay so I just uninstalled the antivirus software and tried again. Same error occurring.

Also, here is the output when I run the execute command:

(venv) marcwitasee@Marcs-Workbook-Air test_netrics % netrics --tb debug execute ping
Name: -
Command: /sbin/ping

Status: Error (Exit code 64)

Result: -

Logged (standard error):

  usage: ping [-AaDdfnoQqRrv] [-c count] [-G sweepmaxsize]
              [-g sweepminsize] [-h sweepincrsize] [-i wait]
              [-l preload] [-M mask | time] [-m ttl] [-p pattern]
              [-S src_addr] [-s packetsize] [-t timeout][-W waittime]
              [-z tos] host
         ping [-AaDdfLnoQqRrv] [-c count] [-I iface] [-i wait]
              [-l preload] [-M mask | time] [-m ttl] [-p pattern] [-S src_addr]
              [-s packetsize] [-T ttl] [-t timeout] [-W waittime]
              [-z tos] mcast-group
  Apple specific options (to be specified before mcast-group or host like all options)
              -b boundif           # bind the socket to the interface
              -k traffic_class     # set traffic class socket option
              -K net_service_type  # set traffic class socket options
              --apple-connect       # call connect(2) in the socket
              --apple-time          # display current time
jesteria commented 1 year ago

That looks like a different error.

Could you try again with the debug run command?

netrics debug run ping
marcwitasee commented 1 year ago

Yeah I was getting the same error with the debug run command. I also get a different error with the debug execute command.

(venv) marcwitasee@Marcs-Workbook-Air test_netrics % netrics debug run ping
Name: ping
Command: /Users/marcwitasee/test_netrics/venv/bin/netrics-ping

Status: Error (Exit code 71)

Result: -

Logged (standard error):

  <5> {"dest": "localhost", "status": "Error", "msg": "host network interface down"}
jesteria commented 1 year ago

Ok sure. (I only asked because the error you posted for the execute command was a different thing – it was telling you to specify a host to ping, such as localhost. But no matter.)

Thanks for the info. This does indeed appear to be specific to the version of ping available on your system – that is, on MacOS – as opposed to on Linux.

That appears to be why you can ping, but it fails in Netrics. In the latter case, we specify arguments which your ping presumably does not accept.

To wit, I presume you can't:

ping -c 1 -w 5 localhost

But it looks like your system would be happy with the following argumentation:

ping -c 1 -t 5 localhost

This could and should be handled. But I will say it's likely not so high a priority as getting things working on our target platform.

We could perhaps do what might be a relatively cheap check in Python against sys.platform == 'darwin' and then confirm that it's not the Linux version of ping with a system call ping -V (which I don't believe MacOS supports).

On Fri, Apr 7, 2023, 10:11 AM Marc Richardson @.***> wrote:

Yeah I was getting the same error with the debug run command. I also get a different error with the debug execute command.

(venv) @.*** test_netrics % netrics debug run ping Name: ping Command: /Users/marcwitasee/test_netrics/venv/bin/netrics-ping

Status: Error (Exit code 71)

Result: -

Logged (standard error):

<5> {"dest": "localhost", "status": "Error", "msg": "host network interface down"} — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you commented.Message ID: ***@***.***>
marcwitasee commented 1 year ago

@jesteria yup I think you are right. I can run the ping command with the -t flag but not the -w flag. And you are also right that the MacOS ping does not support running ping -V.