vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.36k stars 1.51k forks source link

Compose the list of all our dependencies that are update-blocked by `futures` 0.3 #2118

Closed MOZGIII closed 4 years ago

MOZGIII commented 4 years ago

A part of https://github.com/timberio/vector/issues/1142#issuecomment-591418493.

We have dependencies that we couldn't upgrade because they've switched to futures 0.3. We can adopt them now since we've switched to tokio-compat.


Here's a list of dependencies that are themselves blocked on futures 0.1 and/or tokio 0.1. Some crates are blocked specifically on hyper, so we'll sort them out as part of #2117.

Crates that are ready for the upgrade:

Lists are now complete.

We're not in a bad position, we shall proceed with hyper upgrade, followed by the rest of the dependencies.

MOZGIII commented 4 years ago
$ cargo tree -p tokio:0.1.22 -i
tokio v0.1.22
├── hyper v0.12.35
│   ├── hyper-openssl v0.7.1
│   │   └── shiplift v0.6.0
│   │       └── vector v0.9.0 (.../vector)
│   ├── hyper-tls v0.3.2
│   │   ├── reqwest v0.9.24
│   │   │   ├── goauth v0.6.1
│   │   │   │   └── vector v0.9.0 (.../vector) (*)
│   │   │   └── kube v0.16.1
│   │   │       [dev-dependencies]
│   │   │       └── vector v0.9.0 (.../vector) (*)
│   │   └── rusoto_core v0.41.0
│   │       ├── rusoto_cloudwatch v0.41.0
│   │       │   └── vector v0.9.0 (.../vector) (*)
│   │       ├── rusoto_firehose v0.41.0
│   │       │   └── vector v0.9.0 (.../vector) (*)
│   │       ├── rusoto_kinesis v0.41.0
│   │       │   └── vector v0.9.0 (.../vector) (*)
│   │       ├── rusoto_logs v0.41.0
│   │       │   └── vector v0.9.0 (.../vector) (*)
│   │       ├── rusoto_s3 v0.41.0
│   │       │   └── vector v0.9.0 (.../vector) (*)
│   │       ├── rusoto_sts v0.41.0
│   │       │   └── vector v0.9.0 (.../vector) (*)
│   │       └── vector v0.9.0 (.../vector) (*)
│   ├── hyperlocal v0.6.0
│   │   └── shiplift v0.6.0 (*)
│   ├── reqwest v0.9.24 (*)
│   ├── rusoto_core v0.41.0 (*)
│   ├── rusoto_credential v0.41.1
│   │   ├── rusoto_core v0.41.0 (*)
│   │   └── vector v0.9.0 (.../vector) (*)
│   ├── shiplift v0.6.0 (*)
│   └── warp v0.1.21 (git+https://github.com/timberio/warp?branch=0.1.x#017965c627047bf0e53062940e7def127cb21649)
│       └── vector v0.9.0 (.../vector) (*)
│   [dev-dependencies]
│   └── tracing-metrics v0.1.0 (.../vector/lib/tracing-metrics)
│       └── vector v0.9.0 (.../vector) (*)
├── hyperlocal v0.6.0 (*)
├── pulsar v0.3.0
│   └── vector v0.9.0 (.../vector) (*)
├── reqwest v0.9.24 (*)
├── rusoto_core v0.41.0 (*)
├── shiplift v0.6.0 (*)
├── trust-dns v0.17.0
│   └── trust-dns-server v0.17.0
│       [dev-dependencies]
│       └── vector v0.9.0 (.../vector) (*)
│   [dev-dependencies]
│   └── vector v0.9.0 (.../vector) (*)
├── trust-dns-resolver v0.12.0
│   ├── pulsar v0.3.0 (*)
│   ├── trust-dns-server v0.17.0 (*)
│   └── vector v0.9.0 (.../vector) (*)
├── trust-dns-server v0.17.0 (*)
├── vector v0.9.0 (.../vector) (*)
└── warp v0.1.21 (git+https://github.com/timberio/warp?branch=0.1.x#017965c627047bf0e53062940e7def127cb21649) (*)
[dev-dependencies]
└── tracing-metrics v0.1.0 (.../vector/lib/tracing-metrics) (*)
MOZGIII commented 4 years ago
$ cargo tree -i -p futures:0.1.29
futures v0.1.29
├── futures-cpupool v0.1.8
│   └── hyper v0.12.35
│       ├── hyper-openssl v0.7.1
│       │   └── shiplift v0.6.0
│       │       └── vector v0.9.0 (.../vector)
│       ├── hyper-tls v0.3.2
│       │   ├── reqwest v0.9.24
│       │   │   ├── goauth v0.6.1
│       │   │   │   └── vector v0.9.0 (.../vector) (*)
│       │   │   └── kube v0.16.1
│       │   │       [dev-dependencies]
│       │   │       └── vector v0.9.0 (.../vector) (*)
│       │   └── rusoto_core v0.41.0
│       │       ├── rusoto_cloudwatch v0.41.0
│       │       │   └── vector v0.9.0 (.../vector) (*)
│       │       ├── rusoto_firehose v0.41.0
│       │       │   └── vector v0.9.0 (.../vector) (*)
│       │       ├── rusoto_kinesis v0.41.0
│       │       │   └── vector v0.9.0 (.../vector) (*)
│       │       ├── rusoto_logs v0.41.0
│       │       │   └── vector v0.9.0 (.../vector) (*)
│       │       ├── rusoto_s3 v0.41.0
│       │       │   └── vector v0.9.0 (.../vector) (*)
│       │       ├── rusoto_sts v0.41.0
│       │       │   └── vector v0.9.0 (.../vector) (*)
│       │       └── vector v0.9.0 (.../vector) (*)
│       ├── hyperlocal v0.6.0
│       │   └── shiplift v0.6.0 (*)
│       ├── reqwest v0.9.24 (*)
│       ├── rusoto_core v0.41.0 (*)
│       ├── rusoto_credential v0.41.1
│       │   ├── rusoto_core v0.41.0 (*)
│       │   └── vector v0.9.0 (.../vector) (*)
│       ├── shiplift v0.6.0 (*)
│       └── warp v0.1.21 (git+https://github.com/timberio/warp?branch=0.1.x#017965c627047bf0e53062940e7def127cb21649)
│           └── vector v0.9.0 (.../vector) (*)
│       [dev-dependencies]
│       └── tracing-metrics v0.1.0 (.../vector/lib/tracing-metrics)
│           └── vector v0.9.0 (.../vector) (*)
├── futures-timer v0.1.1
│   └── pulsar v0.3.0
│       └── vector v0.9.0 (.../vector) (*)
├── futures-util v0.3.4
│   ├── futures v0.3.4
│   │   ├── file-source v0.1.0 (.../vector/lib/file-source)
│   │   │   └── vector v0.9.0 (.../vector) (*)
│   │   ├── rdkafka v0.23.1
│   │   │   └── vector v0.9.0 (.../vector) (*)
│   │   └── vector v0.9.0 (.../vector) (*)
│   ├── futures-executor v0.3.4
│   │   └── futures v0.3.4 (*)
│   ├── h2 v0.2.2
│   │   └── hyper v0.13.4
│   │       ├── hyper-openssl v0.8.0
│   │       │   └── vector v0.9.0 (.../vector) (*)
│   │       ├── hyper-tls v0.4.1
│   │       │   └── reqwest v0.10.4
│   │       │       [dev-dependencies]
│   │       │       └── vector v0.9.0 (.../vector) (*)
│   │       ├── reqwest v0.10.4 (*)
│   │       └── vector v0.9.0 (.../vector) (*)
│   ├── hyper v0.13.4 (*)
│   ├── reqwest v0.10.4 (*)
│   └── tokio-compat v0.1.5
│       └── vector v0.9.0 (.../vector) (*)
├── futures01 v0.1.29
│   └── snafu v0.4.4
│       └── vector v0.9.0 (.../vector) (*)
├── goauth v0.6.1 (*)
├── h2 v0.1.26
│   └── hyper v0.12.35 (*)
├── http-body v0.1.0
│   └── hyper v0.12.35 (*)
├── hyper v0.12.35 (*)
├── hyper-openssl v0.7.1 (*)
├── hyper-tls v0.3.2 (*)
├── hyperlocal v0.6.0 (*)
├── pulsar v0.3.0 (*)
├── reqwest v0.9.24 (*)
├── rusoto_cloudwatch v0.41.0 (*)
├── rusoto_core v0.41.0 (*)
├── rusoto_credential v0.41.1 (*)
├── rusoto_firehose v0.41.0 (*)
├── rusoto_kinesis v0.41.0 (*)
├── rusoto_logs v0.41.0 (*)
├── rusoto_s3 v0.41.0 (*)
├── rusoto_sts v0.41.0 (*)
├── shiplift v0.6.0 (*)
├── stream-cancel v0.4.4
│   └── vector v0.9.0 (.../vector) (*)
├── tokio v0.1.22
│   ├── hyper v0.12.35 (*)
│   ├── hyperlocal v0.6.0 (*)
│   ├── pulsar v0.3.0 (*)
│   ├── reqwest v0.9.24 (*)
│   ├── rusoto_core v0.41.0 (*)
│   ├── shiplift v0.6.0 (*)
│   ├── trust-dns v0.17.0
│   │   └── trust-dns-server v0.17.0
│   │       [dev-dependencies]
│   │       └── vector v0.9.0 (.../vector) (*)
│   │   [dev-dependencies]
│   │   └── vector v0.9.0 (.../vector) (*)
│   ├── trust-dns-resolver v0.12.0
│   │   ├── pulsar v0.3.0 (*)
│   │   ├── trust-dns-server v0.17.0 (*)
│   │   └── vector v0.9.0 (.../vector) (*)
│   ├── trust-dns-server v0.17.0 (*)
│   ├── vector v0.9.0 (.../vector) (*)
│   └── warp v0.1.21 (git+https://github.com/timberio/warp?branch=0.1.x#017965c627047bf0e53062940e7def127cb21649) (*)
│   [dev-dependencies]
│   └── tracing-metrics v0.1.0 (.../vector/lib/tracing-metrics) (*)
├── tokio-buf v0.1.1
│   ├── http-body v0.1.0 (*)
│   └── hyper v0.12.35 (*)
├── tokio-codec v0.1.1
│   ├── codec v0.1.0 (.../vector/lib/codec)
│   │   └── vector v0.9.0 (.../vector) (*)
│   ├── pulsar v0.3.0 (*)
│   ├── shiplift v0.6.0 (*)
│   ├── tokio v0.1.22 (*)
│   ├── tokio-udp v0.1.5
│   │   ├── tokio v0.1.22 (*)
│   │   ├── trust-dns v0.17.0 (*)
│   │   ├── trust-dns-proto v0.8.0
│   │   │   ├── trust-dns v0.17.0 (*)
│   │   │   ├── trust-dns-resolver v0.12.0 (*)
│   │   │   ├── trust-dns-server v0.17.0 (*)
│   │   │   └── vector v0.9.0 (.../vector) (*)
│   │   │   [dev-dependencies]
│   │   │   └── vector v0.9.0 (.../vector) (*)
│   │   ├── trust-dns-resolver v0.12.0 (*)
│   │   └── trust-dns-server v0.17.0 (*)
│   ├── tokio-uds v0.2.5
│   │   ├── hyperlocal v0.6.0 (*)
│   │   ├── tokio v0.1.22 (*)
│   │   └── vector v0.9.0 (.../vector) (*)
│   └── vector v0.9.0 (.../vector) (*)
├── tokio-compat v0.1.5 (*)
├── tokio-current-thread v0.1.6
│   ├── tokio v0.1.22 (*)
│   └── tokio-compat v0.1.5 (*)
├── tokio-executor v0.1.9
│   ├── hyper v0.12.35 (*)
│   ├── reqwest v0.9.24 (*)
│   ├── tokio v0.1.22 (*)
│   ├── tokio-compat v0.1.5 (*)
│   ├── tokio-current-thread v0.1.6 (*)
│   ├── tokio-reactor v0.1.11
│   │   ├── hyper v0.12.35 (*)
│   │   ├── tokio v0.1.22 (*)
│   │   ├── tokio-compat v0.1.5 (*)
│   │   ├── tokio-process v0.2.4
│   │   │   └── rusoto_credential v0.41.1 (*)
│   │   ├── tokio-signal v0.2.7
│   │   │   ├── tokio-process v0.2.4 (*)
│   │   │   └── vector v0.9.0 (.../vector) (*)
│   │   ├── tokio-tcp v0.1.3
│   │   │   ├── hyper v0.12.35 (*)
│   │   │   ├── tokio v0.1.22 (*)
│   │   │   ├── trust-dns v0.17.0 (*)
│   │   │   ├── trust-dns-proto v0.8.0 (*)
│   │   │   ├── trust-dns-resolver v0.12.0 (*)
│   │   │   └── trust-dns-server v0.17.0 (*)
│   │   ├── tokio-udp v0.1.5 (*)
│   │   ├── tokio-uds v0.2.5 (*)
│   │   ├── trust-dns-proto v0.8.0 (*)
│   │   └── trust-dns-server v0.17.0 (*)
│   ├── tokio-signal v0.2.7 (*)
│   ├── tokio-threadpool v0.1.17
│   │   ├── hyper v0.12.35 (*)
│   │   ├── reqwest v0.9.24 (*)
│   │   ├── tokio v0.1.22 (*)
│   │   ├── tokio-fs v0.1.6
│   │   │   └── tokio v0.1.22 (*)
│   │   └── warp v0.1.21 (git+https://github.com/timberio/warp?branch=0.1.x#017965c627047bf0e53062940e7def127cb21649) (*)
│   ├── tokio-timer v0.2.12
│   │   ├── hyper v0.12.35 (*)
│   │   ├── reqwest v0.9.24 (*)
│   │   ├── rusoto_core v0.41.0 (*)
│   │   ├── rusoto_credential v0.41.1 (*)
│   │   ├── tokio v0.1.22 (*)
│   │   ├── tokio-compat v0.1.5 (*)
│   │   ├── tokio-retry v0.2.0
│   │   │   └── vector v0.9.0 (.../vector) (*)
│   │   ├── tokio01-test v0.1.1
│   │   │   [dev-dependencies]
│   │   │   └── vector v0.9.0 (.../vector) (*)
│   │   ├── tower-limit v0.1.1
│   │   │   └── tower v0.1.1
│   │   │       └── vector v0.9.0 (.../vector) (*)
│   │   ├── tower-retry v0.1.0
│   │   │   └── tower v0.1.1 (*)
│   │   ├── tower-timeout v0.1.1
│   │   │   └── tower v0.1.1 (*)
│   │   ├── trust-dns-proto v0.8.0 (*)
│   │   └── trust-dns-server v0.17.0 (*)
│   ├── tokio01-test v0.1.1 (*)
│   ├── tower-buffer v0.1.2
│   │   └── tower v0.1.1 (*)
│   ├── trust-dns-proto v0.8.0 (*)
│   ├── trust-dns-resolver v0.12.0 (*)
│   └── trust-dns-server v0.17.0 (*)
├── tokio-fs v0.1.6 (*)
├── tokio-io v0.1.12
│   ├── h2 v0.1.26 (*)
│   ├── hyper v0.12.35 (*)
│   ├── hyper-openssl v0.7.1 (*)
│   ├── hyper-tls v0.3.2 (*)
│   ├── hyperlocal v0.6.0 (*)
│   ├── reqwest v0.9.24 (*)
│   ├── shiplift v0.6.0 (*)
│   ├── tokio v0.1.22 (*)
│   ├── tokio-codec v0.1.1 (*)
│   ├── tokio-fs v0.1.6 (*)
│   ├── tokio-openssl v0.3.0
│   │   ├── hyper-openssl v0.7.1 (*)
│   │   └── vector v0.9.0 (.../vector) (*)
│   ├── tokio-process v0.2.4 (*)
│   ├── tokio-reactor v0.1.11 (*)
│   ├── tokio-signal v0.2.7 (*)
│   ├── tokio-tcp v0.1.3 (*)
│   ├── tokio-udp v0.1.5 (*)
│   ├── tokio-uds v0.2.5 (*)
│   ├── tower-util v0.1.0
│   │   ├── tower v0.1.1 (*)
│   │   └── tracing-tower v0.1.0 (git+https://github.com/tokio-rs/tracing?rev=65547d8809fcc726b8187db85b23c42e32ef5dce#65547d8809fcc726b8187db85b23c42e32ef5dce)
│   │       └── vector v0.9.0 (.../vector) (*)
│   ├── trust-dns-proto v0.8.0 (*)
│   ├── trust-dns-server v0.17.0 (*)
│   └── warp v0.1.21 (git+https://github.com/timberio/warp?branch=0.1.x#017965c627047bf0e53062940e7def127cb21649) (*)
├── tokio-openssl v0.3.0 (*)
├── tokio-process v0.2.4 (*)
├── tokio-reactor v0.1.11 (*)
├── tokio-retry v0.2.0 (*)
├── tokio-signal v0.2.7 (*)
├── tokio-sync v0.1.7
│   ├── hotmic v0.8.2
│   │   ├── tracing-metrics v0.1.0 (.../vector/lib/tracing-metrics) (*)
│   │   └── vector v0.9.0 (.../vector) (*)
│   ├── tokio v0.1.22 (*)
│   ├── tokio-reactor v0.1.11 (*)
│   ├── tower-buffer v0.1.2 (*)
│   ├── tower-limit v0.1.1 (*)
│   └── tower-test v0.1.0
│       [dev-dependencies]
│       └── vector v0.9.0 (.../vector) (*)
├── tokio-tcp v0.1.3 (*)
├── tokio-threadpool v0.1.17 (*)
├── tokio-timer v0.2.12 (*)
├── tokio-udp v0.1.5 (*)
├── tokio-uds v0.2.5 (*)
├── tokio01-test v0.1.1 (*)
├── tower v0.1.1 (*)
├── tower-buffer v0.1.2 (*)
├── tower-discover v0.1.0
│   └── tower v0.1.1 (*)
├── tower-layer v0.1.0
│   ├── tower v0.1.1 (*)
│   ├── tower-buffer v0.1.2 (*)
│   ├── tower-limit v0.1.1 (*)
│   ├── tower-load-shed v0.1.0
│   │   └── tower v0.1.1 (*)
│   ├── tower-retry v0.1.0 (*)
│   ├── tower-timeout v0.1.1 (*)
│   ├── tower-util v0.1.0 (*)
│   └── tracing-tower v0.1.0 (git+https://github.com/tokio-rs/tracing?rev=65547d8809fcc726b8187db85b23c42e32ef5dce#65547d8809fcc726b8187db85b23c42e32ef5dce) (*)
├── tower-limit v0.1.1 (*)
├── tower-load-shed v0.1.0 (*)
├── tower-retry v0.1.0 (*)
├── tower-service v0.2.0
│   ├── tower v0.1.1 (*)
│   ├── tower-buffer v0.1.2 (*)
│   ├── tower-discover v0.1.0 (*)
│   ├── tower-layer v0.1.0 (*)
│   ├── tower-limit v0.1.1 (*)
│   ├── tower-load-shed v0.1.0 (*)
│   ├── tower-retry v0.1.0 (*)
│   ├── tower-test v0.1.0 (*)
│   ├── tower-timeout v0.1.1 (*)
│   ├── tower-util v0.1.0 (*)
│   └── tracing-tower v0.1.0 (git+https://github.com/tokio-rs/tracing?rev=65547d8809fcc726b8187db85b23c42e32ef5dce#65547d8809fcc726b8187db85b23c42e32ef5dce) (*)
├── tower-test v0.1.0 (*)
├── tower-timeout v0.1.1 (*)
├── tower-util v0.1.0 (*)
├── tracing-futures v0.2.0 (git+https://github.com/tokio-rs/tracing?rev=65547d8809fcc726b8187db85b23c42e32ef5dce#65547d8809fcc726b8187db85b23c42e32ef5dce)
│   └── tracing-tower v0.1.0 (git+https://github.com/tokio-rs/tracing?rev=65547d8809fcc726b8187db85b23c42e32ef5dce#65547d8809fcc726b8187db85b23c42e32ef5dce) (*)
├── tracing-futures v0.2.1
│   └── vector v0.9.0 (.../vector) (*)
│   [dev-dependencies]
│   └── tracing-metrics v0.1.0 (.../vector/lib/tracing-metrics) (*)
├── tracing-tower v0.1.0 (git+https://github.com/tokio-rs/tracing?rev=65547d8809fcc726b8187db85b23c42e32ef5dce#65547d8809fcc726b8187db85b23c42e32ef5dce) (*)
├── trust-dns v0.17.0 (*)
├── trust-dns-proto v0.8.0 (*)
├── trust-dns-resolver v0.12.0 (*)
├── trust-dns-server v0.17.0 (*)
├── vector v0.9.0 (.../vector) (*)
├── want v0.2.0
│   └── hyper v0.12.35 (*)
└── warp v0.1.21 (git+https://github.com/timberio/warp?branch=0.1.x#017965c627047bf0e53062940e7def127cb21649) (*)
[dev-dependencies]
└── tracing-metrics v0.1.0 (.../vector/lib/tracing-metrics) (*)
binarylogic commented 4 years ago

@LucioFranco can you update this issue to reflect your work? Please add any missing dependencies and turn the list into checkboxes.

LucioFranco commented 4 years ago

Okay quick update on upgrade progress.

So far there have been two PRs merged #2294 and #2362.

Currently the process we are going with is to upgrade sink util in as small units of work as possible to ensure we don't introduce any regressions. Once, the util have been upgraded it should be quite trivial to upgrade all our sinks.

Here is a list for upgrade progress in relation to sinks which include the most abstractions and the most upgrade work in general.

I've broken this up into individual work that is grouped by logical type. Most of the sink/util work needs to be done step by step, but once we have most of that out with two versions aka upgraded and not upgraded. We can slowly start to migrate sinks to use the updated version without much worry about interfering with each others work.

All transforms and sources should be able to be upgraded individually since they don't have many shared abstractions but once I get through util upgrade I'll map out how we can tackle it.

Notes

You may notice that sink isn't included in this list, for now I'd like to avoid upgrading our sink trait or internal batching machinery until we have figured out in #2122.

fanatid commented 4 years ago

shiplift (docker source) still use futures 0.1 and no progress with update.

@ktff how you think, is it possible use bollard instead shiplift? Will this need a lot of work? (https://github.com/timberio/vector/pull/787)

ktff commented 4 years ago

It shouldn't require too much. The configuration part will probably be mostly effected, followed by usages of structs that we get from bollard. The api is quite similar, as they are interfacing with the same docker api. There could be some feature mismatch, but all of them should be addressable from our side.

So, yes, moving to bollard should certainly be possible, as a plus bollard has one optimization feature that shiplift doesn't.

binarylogic commented 4 years ago

Closing in favor of the above linked issues.