BerlinVagrant / vagrant-dns

A plugin to manage DNS records for vagrant environments
MIT License
490 stars 50 forks source link

Problems with async-io ^1.3? #59

Closed mpdude closed 6 years ago

mpdude commented 6 years ago

I just upgraded to Vagrant 2.0.3 and since then had issues starting VMs and/or resolving DNS names provided by vagrant-dns. From skimming Vagrant debug output, I am under the impression that this plugin might have something to do with it. But I am not a Ruby developer, so I might be entirely wrong.

So, in order to fix it, I first tried vagrant plugin expunge, which re-installed all plugins and dependencies as follows:

Fetching: vagrant-bindfs-1.1.0.gem (100%)
Fetching: vagrant-disksize-0.1.2.gem (100%)
Fetching: daemons-1.2.6.gem (100%)
Fetching: nio4r-2.3.0.gem (100%)
Building native extensions.  This could take a while...
Fetching: hitimes-1.2.6.gem (100%)
Building native extensions.  This could take a while...
Fetching: timers-4.1.2.gem (100%)
Fetching: async-1.3.0.gem (100%)
Fetching: async-io-1.4.0.gem (100%)
Fetching: async-dns-1.0.0.gem (100%)
Fetching: rubydns-2.0.1.gem (100%)
Fetching: vagrant-dns-2.1.0.gem (100%)
Fetching: vagrant-hostsupdater-1.0.2.gem (100%)
Fetching: bundler-1.16.1.gem (100%)
Fetching: vagrant-triggers-0.5.3.gem (100%)
Installed the plugin 'vagrant-bindfs (1.1.0)'!
Installing the 'vagrant-disksize' plugin. This can take a few minutes...
Installed the plugin 'vagrant-disksize (0.1.2)'!
Installing the 'vagrant-dns' plugin. This can take a few minutes...

Installed the plugin 'vagrant-dns (2.1.0)'!
Installing the 'vagrant-hostsupdater' plugin. This can take a few minutes...
Installed the plugin 'vagrant-hostsupdater (1.0.2)'!
Installing the 'vagrant-triggers' plugin. This can take a few minutes...
Installed the plugin 'vagrant-triggers (0.5.3)'!

Then, vagrant dns --start --ontop gives me:

/Users/mp/.vagrant.d/gems/2.4.3/gems/async-dns-1.0.0/lib/async/dns/server.rb:128:in `block in setup_handlers': undefined method `socket_type' for #<Async::IO::HostEndpoint:0x0000000102870f70> (NoMethodError)
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-io-1.4.0/lib/async/io/endpoint.rb:66:in `block in each'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-io-1.4.0/lib/async/io/endpoint.rb:62:in `each'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-io-1.4.0/lib/async/io/endpoint.rb:62:in `each'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-dns-1.0.0/lib/async/dns/server.rb:127:in `setup_handlers'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-dns-1.0.0/lib/async/dns/server.rb:109:in `run'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/rubydns-2.0.1/lib/rubydns.rb:39:in `run_server'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/service.rb:38:in `block in run!'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/application.rb:270:in `block in start_proc'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/application.rb:279:in `start_proc'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/application.rb:300:in `start'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/controller.rb:56:in `run'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons.rb:197:in `block in run_proc'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/cmdline.rb:92:in `catch_exceptions'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons.rb:196:in `run_proc'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/service.rb:31:in `run!'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/service.rb:17:in `start!'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/command.rb:105:in `manage_service'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/command.rb:77:in `execute'
    from /opt/vagrant/embedded/gems/2.0.3/gems/vagrant-2.0.3/lib/vagrant/cli.rb:46:in `execute'
    from /opt/vagrant/embedded/gems/2.0.3/gems/vagrant-2.0.3/lib/vagrant/environment.rb:269:in `cli'
    from /opt/vagrant/embedded/gems/2.0.3/gems/vagrant-2.0.3/bin/vagrant:151:in `<main>'

This leads us to async-dns, where I found https://github.com/socketry/async-dns/issues/6.

So – I am dropping this here to 1) let you know and 2) kindly as whether there is anything you can to to quickly work around this, for example by forcing the async-dns dependency < 1.3?

Thanks!

mpdude commented 6 years ago

Downgrading vagrant-dns to prior 2.0 makes the problem go away, as then it used celluloid instead of async-dns.

$ vagrant plugin install vagrant-dns --plugin-version 1.1.0
Installing the 'vagrant-dns --version '1.1.0'' plugin. This can take a few minutes...
Fetching: timers-4.0.4.gem (100%)
Fetching: celluloid-0.16.0.gem (100%)
Fetching: celluloid-io-0.16.2.gem (100%)
Fetching: rubydns-1.0.3.gem (100%)
Fetching: vagrant-dns-1.1.0.gem (100%)
Successfully uninstalled async-1.3.0
Successfully uninstalled async-dns-1.0.0
Successfully uninstalled async-io-1.4.0
Removing rubydns-check
Successfully uninstalled rubydns-2.0.1
Successfully uninstalled timers-4.1.2
Successfully uninstalled vagrant-dns-2.1.0
Installed the plugin 'vagrant-dns (1.1.0)'!

$ vagrant dns --start --ontop
I, [2018-03-19T17:35:27.366088 #17903]  INFO -- : Starting RubyDNS server (v1.0.3)...
I, [2018-03-19T17:35:27.366149 #17903]  INFO -- : <> Listening on udp:127.0.0.1:5300
^C
D, [2018-03-19T17:35:32.605919 #17903] DEBUG -- : Terminating 2 actors...
W, [2018-03-19T17:35:32.606321 #17903]  WARN -- : Terminating task: type=:call, meta={:method_name=>:run}, status=:iowait
    Celluloid::TaskFiber backtrace unavailable. Please try `Celluloid.task_class = Celluloid::TaskThread` if you need backtraces here.

$ vagrant plugin install vagrant-dns --plugin-version=2.1.0
Installing the 'vagrant-dns --version '2.1.0'' plugin. This can take a few minutes...
Fetching: timers-4.1.2.gem (100%)
Fetching: async-1.3.0.gem (100%)
Fetching: async-io-1.4.0.gem (100%)
Fetching: async-dns-1.0.0.gem (100%)
Fetching: rubydns-2.0.1.gem (100%)
Fetching: vagrant-dns-2.1.0.gem (100%)
Successfully uninstalled celluloid-0.16.0
Successfully uninstalled celluloid-io-0.16.2
Removing rubydns-check
Successfully uninstalled rubydns-1.0.3
Successfully uninstalled timers-4.0.4
Successfully uninstalled vagrant-dns-1.1.0
Installed the plugin 'vagrant-dns (2.1.0)'!

$ vagrant dns --start --ontop
/Users/mp/.vagrant.d/gems/2.4.3/gems/async-dns-1.0.0/lib/async/dns/server.rb:128:in `block in setup_handlers': undefined method `socket_type' for #<Async::IO::HostEndpoint:0x0000000101ca97c8> (NoMethodError)
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-io-1.4.0/lib/async/io/endpoint.rb:66:in `block in each'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-io-1.4.0/lib/async/io/endpoint.rb:62:in `each'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-io-1.4.0/lib/async/io/endpoint.rb:62:in `each'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-dns-1.0.0/lib/async/dns/server.rb:127:in `setup_handlers'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-dns-1.0.0/lib/async/dns/server.rb:109:in `run'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/rubydns-2.0.1/lib/rubydns.rb:39:in `run_server'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/service.rb:38:in `block in run!'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/application.rb:270:in `block in start_proc'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/application.rb:279:in `start_proc'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/application.rb:300:in `start'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/controller.rb:56:in `run'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons.rb:197:in `block in run_proc'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/cmdline.rb:92:in `catch_exceptions'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons.rb:196:in `run_proc'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/service.rb:31:in `run!'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/service.rb:17:in `start!'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/command.rb:105:in `manage_service'
    from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/command.rb:77:in `execute'
    from /opt/vagrant/embedded/gems/2.0.3/gems/vagrant-2.0.3/lib/vagrant/cli.rb:46:in `execute'
    from /opt/vagrant/embedded/gems/2.0.3/gems/vagrant-2.0.3/lib/vagrant/environment.rb:269:in `cli'
    from /opt/vagrant/embedded/gems/2.0.3/gems/vagrant-2.0.3/bin/vagrant:151:in `<main>'
fnordfish commented 6 years ago

Hi. That’s pretty weird.
Sorry, but I won’t be able to look into that for at least another 2 weeks.
The only thing I can think of right now is: have you also upgraded vagrant? This might lead to loading of not matching gems. If so, try to delete “vagrant.d/gems” and reinstall all plugins :|

--
Robert Schulze

From: Matthias Pigulla notifications@github.com(mailto:notifications@github.com) Reply: BerlinVagrant/vagrant-dns reply@reply.github.com(mailto:reply@reply.github.com) Date: 20. March 2018 at 02:39:14 To: BerlinVagrant/vagrant-dns vagrant-dns@noreply.github.com(mailto:vagrant-dns@noreply.github.com) CC: Subscribed subscribed@noreply.github.com(mailto:subscribed@noreply.github.com) Subject: Re: [BerlinVagrant/vagrant-dns] Problems with async-io ^1.3? (#59)

Downgrading vagrant-dns to prior 2.0 makes the problem go away, as then it used celluloid instead of async-dns.

$ vagrant plugin install vagrant-dns --plugin-version 1.1.0 Installing the 'vagrant-dns --version '1.1.0'' plugin. This can take a few minutes... Fetching: timers-4.0.4.gem (100%) Fetching: celluloid-0.16.0.gem (100%) Fetching: celluloid-io-0.16.2.gem (100%) Fetching: rubydns-1.0.3.gem (100%) Fetching: vagrant-dns-1.1.0.gem (100%) Successfully uninstalled async-1.3.0 Successfully uninstalled async-dns-1.0.0 Successfully uninstalled async-io-1.4.0 Removing rubydns-check Successfully uninstalled rubydns-2.0.1 Successfully uninstalled timers-4.1.2 Successfully uninstalled vagrant-dns-2.1.0 Installed the plugin 'vagrant-dns (1.1.0)'! $ vagrant dns --start --ontop I, [2018-03-19T17:35:27.366088 #17903] INFO -- : Starting RubyDNS server (v1.0.3)... I, [2018-03-19T17:35:27.366149 #17903] INFO -- : <> Listening on udp:127.0.0.1:5300 ^C D, [2018-03-19T17:35:32.605919 #17903] DEBUG -- : Terminating 2 actors... W, [2018-03-19T17:35:32.606321 #17903] WARN -- : Terminating task: type=:call, meta={:method_name=>:run}, status=:iowait Celluloid::TaskFiber backtrace unavailable. Please try Celluloid.task_class = Celluloid::TaskThread if you need backtraces here. $ vagrant plugin install vagrant-dns --plugin-version=2.1.0 Installing the 'vagrant-dns --version '2.1.0'' plugin. This can take a few minutes... Fetching: timers-4.1.2.gem (100%) Fetching: async-1.3.0.gem (100%) Fetching: async-io-1.4.0.gem (100%) Fetching: async-dns-1.0.0.gem (100%) Fetching: rubydns-2.0.1.gem (100%) Fetching: vagrant-dns-2.1.0.gem (100%) Successfully uninstalled celluloid-0.16.0 Successfully uninstalled celluloid-io-0.16.2 Removing rubydns-check Successfully uninstalled rubydns-1.0.3 Successfully uninstalled timers-4.0.4 Successfully uninstalled vagrant-dns-1.1.0 Installed the plugin 'vagrant-dns (2.1.0)'! $ vagrant dns --start --ontop /Users/mp/.vagrant.d/gems/2.4.3/gems/async-dns-1.0.0/lib/async/dns/server.rb:128:in block in setup_handlers': undefined methodsocket_type' for # (NoMethodError) from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-io-1.4.0/lib/async/io/endpoint.rb:66:in block in each' from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-io-1.4.0/lib/async/io/endpoint.rb:62:ineach' from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-io-1.4.0/lib/async/io/endpoint.rb:62:in each' from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-dns-1.0.0/lib/async/dns/server.rb:127:insetup_handlers' from /Users/mp/.vagrant.d/gems/2.4.3/gems/async-dns-1.0.0/lib/async/dns/server.rb:109:in run' from /Users/mp/.vagrant.d/gems/2.4.3/gems/rubydns-2.0.1/lib/rubydns.rb:39:inrun_server' from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/service.rb:38:in block in run!' from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/application.rb:270:inblock in start_proc' from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/application.rb:279:in start_proc' from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/application.rb:300:instart' from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/controller.rb:56:in run' from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons.rb:197:inblock in run_proc' from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/cmdline.rb:92:in catch_exceptions' from /Users/mp/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons.rb:196:inrun_proc' from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/service.rb:31:in run!' from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/service.rb:17:instart!' from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/command.rb:105:in manage_service' from /Users/mp/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/command.rb:77:inexecute' from /opt/vagrant/embedded/gems/2.0.3/gems/vagrant-2.0.3/lib/vagrant/cli.rb:46:in execute' from /opt/vagrant/embedded/gems/2.0.3/gems/vagrant-2.0.3/lib/vagrant/environment.rb:269:incli' from /opt/vagrant/embedded/gems/2.0.3/gems/vagrant-2.0.3/bin/vagrant:151:in `

'

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub(https://github.com/BerlinVagrant/vagrant-dns/issues/59#issuecomment-374278994), or mute the thread(https://github.com/notifications/unsubscribe-auth/AAAyJ6mlt1TEO0PyrjKSoRPl81JKJSTxks5tf98ygaJpZM4SwTeU).

mpdude commented 6 years ago

Yes, it started with the Vagrant update to 2.0.3. As far as I can tell, vagrant plugin expunge should have reinstalled all plugins anyway.

To me it looks that async-io inadvertently changed their API.

For the time being, installing an older version of vagrant-dns is a workaround.

ac21 commented 6 years ago

+1; same issue; Vagrant 2.0.3 and vagrant-dns 2.1.0 are not working together with the same error:

% vagrant dns --start --ontop
/Users/andrew/.vagrant.d/gems/2.4.3/gems/async-dns-1.0.0/lib/async/dns/server.rb:128:in `block in setup_handlers': undefined method `socket_type' for #<Async::IO::HostEndpoint:0x000000010833a460> (NoMethodError)
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/async-io-1.6.1/lib/async/io/endpoint.rb:86:in `block in each'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/async-io-1.6.1/lib/async/io/endpoint.rb:85:in `each'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/async-io-1.6.1/lib/async/io/endpoint.rb:85:in `each'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/async-dns-1.0.0/lib/async/dns/server.rb:127:in `setup_handlers'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/async-dns-1.0.0/lib/async/dns/server.rb:109:in `run'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/rubydns-2.0.1/lib/rubydns.rb:39:in `run_server'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/service.rb:38:in `block in run!'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/application.rb:270:in `block in start_proc'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/application.rb:279:in `start_proc'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/application.rb:300:in `start'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/controller.rb:56:in `run'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons.rb:197:in `block in run_proc'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons/cmdline.rb:92:in `catch_exceptions'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/daemons-1.2.6/lib/daemons.rb:196:in `run_proc'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/service.rb:31:in `run!'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/service.rb:17:in `start!'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/command.rb:105:in `manage_service'
    from /Users/andrew/.vagrant.d/gems/2.4.3/gems/vagrant-dns-2.1.0/lib/vagrant-dns/command.rb:77:in `execute'
    from /opt/vagrant/embedded/gems/2.0.3/gems/vagrant-2.0.3/lib/vagrant/cli.rb:46:in `execute'
    from /opt/vagrant/embedded/gems/2.0.3/gems/vagrant-2.0.3/lib/vagrant/environment.rb:269:in `cli'
    from /opt/vagrant/embedded/gems/2.0.3/gems/vagrant-2.0.3/bin/vagrant:151:in `<main>'

replacing with vagrant plugin install vagrant-dns --plugin-version 1.1.0 is a valid workaround

ioquatix commented 6 years ago

I apologise for this issue. It's my fault.

I will investigate the issue and report back.

Yes, there were some "internal" API changes, but it did require both gems to be updated. Perhaps I should have bumped major version, but I considered it to be implementation detail.

ioquatix commented 6 years ago

I apologies again. I found out that there was supposed to be a release 1.1.0 of async-dns which is compatible with the latest async-io but for some reason I never got around to releasing it. I will check now and release it.

ioquatix commented 6 years ago

Okay, I've just released async-dns 1.1.0 which should be compatible with async-io ~> 1.3.0. Can you please check that this is working with vagrant-dns and if there are any issues let me know and I will sort it out.

fnordfish commented 6 years ago

Thanks for jumping in @ioquatix!

Just uninstalled vagrant-dns, which gave me:

Uninstalling the 'vagrant-dns' plugin...
Successfully uninstalled async-1.2.0
Successfully uninstalled async-dns-1.0.0
Successfully uninstalled async-io-1.0.0
Successfully uninstalled daemons-1.2.6
Successfully uninstalled hitimes-1.2.6
Successfully uninstalled nio4r-2.2.0
Removing rubydns-check
Successfully uninstalled rubydns-2.0.1
Successfully uninstalled timers-4.1.2
Successfully uninstalled vagrant-dns-2.1.0

And re-installed:

Installing the 'vagrant-dns' plugin. This can take a few minutes...
Fetching: daemons-1.2.6.gem (100%)
Fetching: nio4r-2.3.0.gem (100%)
Building native extensions.  This could take a while...
Fetching: hitimes-1.2.6.gem (100%)
Building native extensions.  This could take a while...
Fetching: timers-4.1.2.gem (100%)
Fetching: async-1.5.0.gem (100%)
Fetching: async-io-1.7.0.gem (100%)
Fetching: async-dns-1.1.0.gem (100%)
Fetching: rubydns-2.0.1.gem (100%)
Fetching: vagrant-dns-2.1.0.gem (100%)
fnordfish commented 6 years ago

@mpdude can you confirm, that re-installation will fix the dependency issue?

mpdude commented 6 years ago

Works for me! 👍

Thank you @ioquatix and @fnordfish!