Remove dependency on tokio 0.1

lukesteensen commented 4 years ago

This is a prerequisite for removing tokio-compat and using some nice things like the #[tokio::test] macro.

First, we need to upgrade dependencies that rely on tokio 0.1 (some of these are in progress):

$ cargo tree -i -p tokio:0.1.22          
tokio v0.1.22                                                   
├── hyper v0.12.35                                              
│   ├── hyper-openssl v0.7.1                                    
│   │   └── shiplift v0.6.0                                     
│   │       └── vector v0.10.0 (/home/luke/code/vector)         
│   ├── hyper-tls v0.3.2                                        
│   │   └── reqwest v0.9.24                                     
│   │       └── vector v0.10.0 (/home/luke/code/vector)         
│   ├── hyperlocal v0.6.0                                       
│   │   └── shiplift v0.6.0 (*)                                 
│   ├── reqwest v0.9.24 (*)                                     
│   └── shiplift v0.6.0 (*)                                     
├── hyperlocal v0.6.0 (*)                                       
├── pulsar v0.3.0                                               
│   └── vector v0.10.0 (/home/luke/code/vector)                 
├── reqwest v0.9.24 (*)                                         
├── shiplift v0.6.0 (*)                                         
├── trust-dns-resolver v0.12.0                                  
│   └── pulsar v0.3.0 (*)                                       
└── vector v0.10.0 (/home/luke/code/vector)

Second, we need to remove tokio 0.1 types from our sources, sinks, and transforms. These consist mostly of timers and IO types like sockets that we should switch to the equivalent tokio 0.2 types. Where relevant and not too invasive, this can also be a good time to do some light updates away from futures 0.1 combinators and towards async/await. The following is a rough list of files that import tokio01:

[ ] src/topology/fanout.rs
[ ] src/buffers/mod.rs
[ ] src/topology/config/watcher.rs
[ ] src/topology/mod.rs
[ ] src/topology/builder.rs
[x] src/shutdown.rs (ktff, #3106)
[ ] src/runtime.rs
[ ] src/sinks/socket.rs (Kirill, #3188)
[x] src/sinks/util/test.rs (Kirill, #3038)
[ ] src/sinks/util/retries.rs
[ ] src/sinks/util/sink.rs
[ ] src/sinks/util/tcp.rs (Kirill, #3188)
[x] src/sinks/util/unix.rs (Kirill, #3095)
[ ] src/sinks/util/udp.rs
[ ] src/sinks/util/service.rs
[ ] src/sinks/util/buffer/metrics.rs
[ ] src/sinks/util/buffer/mod.rs
[ ] src/sinks/clickhouse.rs
[x] src/sinks/statsd.rs (Kirill, #3095)
[x] src/sinks/gcp/mod.rs (Kirill, #2993)
[x] src/async_read.rs (Kirill, #3095)
[ ] src/test_util.rs
[ ] src/sources/journald.rs
[ ] src/transforms/aws_ec2_metadata.rs
[ ] src/sources/util/tcp.rs (Kirill, #3188)
[x] src/sources/util/unix.rs (Kirill, #3095)
[ ] src/sources/prometheus/mod.rs
[x] src/transforms/util/runtime_transform.rs (Kirill, #3065)
[ ] src/tls/maybe_tls.rs (Kirill, #3188)
[ ] src/tls/mod.rs (Kirill, #3188)
[ ] src/tls/incoming.rs (Kirill, #3188)
[x] src/sources/syslog.rs (Kirill, #3095)
[ ] src/tls/outgoing.rs (Kirill, #3188)
[ ] src/sources/docker.rs
[x] src/sources/vector.rs (Kirill, #3095)
[ ] src/sources/file/line_agg.rs
[x] src/sources/socket/mod.rs (Kirill, #3095)
[x] src/sources/statsd/mod.rs (Kirill, #3095)
[x] src/stream.rs (#3065)
[ ] src/sources/file/mod.rs
[x] src/sources/socket/udp.rs (Kirill, #3095)
[x] src/sources/socket/unix.rs (Kirill, #3095)
[x] benches/files.rs (Kirill, #3095)
[x] benches/http.rs (ktff, #3107)
[x] tests/tcp.rs (Kirill, #3095)
[x] tests/syslog.rs (Kirill, #3095)
[ ] Add tokio 0.1 to cargo-deny

This work should be very easy to work on in parallel, so feel free to grab a dependency or a specific file and go to work on a PR.

binarylogic commented 4 years ago

@kirillt @ktff would you mind putting your name by each file before you begin work? This will ensure there are no conflicts. I'm happy to split this out into separate issues if you think it'll be easier, but I think it'll add a significant amount of noise.

fanatid commented 4 years ago

New dependency tree (https://github.com/timberio/vector/commit/ae53b889406942dd404578fc84d2675ea2e282ac):

$ cargo tree -i -p tokio:0.1.22
tokio v0.1.22
├── pulsar v0.3.0
│   └── vector v0.10.0 (/home/kirill/projects/vector)
├── trust-dns-resolver v0.12.0
│   └── pulsar v0.3.0 (*)
└── vector v0.10.0 (/home/kirill/projects/vector)

After pulsar update (https://github.com/timberio/vector/pull/2977) only vector will depend from tokio:0.1.22.

Edit: pulsar merged (https://github.com/timberio/vector/tree/ccbb802c010c6109240a14be44b3c30dbae73564):

$ cargo tree -i -p tokio:0.1.22
tokio v0.1.22
└── vector v0.10.0 (/home/kirill/projects/vector)

fanatid commented 4 years ago

@ktff I'll start on updating codec, roughly this files will be affected:

$ grep 'codec::' -rl src/
src/sinks/statsd.rs
src/sinks/util/tcp.rs
src/sinks/util/unix.rs
src/test_util.rs
src/sources/vector.rs
src/sources/statsd/mod.rs
src/sources/http.rs
src/sources/syslog.rs
src/sources/socket/unix.rs
src/sources/socket/mod.rs
src/sources/util/tcp.rs
src/sources/util/unix.rs

ktff commented 4 years ago

@fanatid I'll start with src/shutdown.rs.

ktff commented 4 years ago

@fanatid I'll start with benches/http.rs

fanatid commented 4 years ago

@ktff I'll start work on updating MaybeTlsStream and related things (currently on top of #3095, will rebase when will be merged).

fanatid commented 4 years ago

@ktff I'm going remove https://github.com/timberio/vector/blob/6a6c390b7e7af2b977ff088245e94e31cefb7640/src/stream.rs#L40 and update affected code.

fanatid commented 4 years ago

@ktff looks like not a lot of tokio01 usage left, I do not think that it cost divide files, so I'll take rest.

ktff commented 4 years ago

@fanatid how is this going? Need any extra hands?

fanatid commented 4 years ago

Need only update my previous TLS PR #3188 and we should be able remove tokio:0.1. Should finish soon, no need help right now I think. Thanks!

vectordotdev / vector

Remove dependency on tokio 0.1 #2945