twmb/franz-go
### [`v1.5.2`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v152)
[Compare Source](https://togithub.com/twmb/franz-go/compare/v1.5.1...v1.5.2)
\===
This tiny release contains another fix for `RecordReader`, specifically for
regex parsing, and changes the behavior of `MarkCommitRecords` to its original
behavior.
- For `RecordReader`, the regex type was not properly registered everywhere
internally. This resulted in `%v{re#...#}\n` trying to parse a newline as
part of the regex, rather than as a delimiter. This is fixed and a test case
added. Note that this feature is niche *and* new, so this is a very minor bug.
- For `MarkCommitRecords`, v1.3.3 changed the behavior to allow rewinds. This
is difficult to reason about and can result in bugs. More likely, you just
want to periodically mark to move forward; the behavior now is more in line
with what people expect.
- [`ff5a3ed`](https://togithub.com/twmb/franz-go/commit/ff5a3ed) MarkCommitRecords: forbid rewinds
- [`41284b3`](https://togithub.com/twmb/franz-go/commit/41284b3) RecordReader: fix regex reading even more
### [`v1.5.1`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v151)
[Compare Source](https://togithub.com/twmb/franz-go/compare/v1.5.0...v1.5.1)
\===
This release fixes a minor bug in `RecordReader`, and has a behavior change for
polling records (and a corresponding tiny new helper API for this).
For the bugfix, `RecordReader` did not always properly return
`io.ErrUnexpectedEOF`. We now return it more properly and add tests for the
missing edge cases.
For the behavior change, we now inject an error into `PollFetches` and
`PollRecords` if the user context is canceled. Previously, we would just quit
the poll and return. This change introduces a new corresponding function,
`Fetches.Err() error`. The thought here is that now we have more injected
errors, so `Fetches.Err` can make it easier to check various errors.
- [`cbc8962`](https://togithub.com/twmb/franz-go/commit/cbc8962) **behavior change** Poll{Records,Fetches}: inject an error for canceled context
- [#163](https://togithub.com/twmb/franz-go/pull/163) docs: updates StickyBalancer godoc (thanks [@Streppel](https://togithub.com/Streppel))
- [`2018d20`](https://togithub.com/twmb/franz-go/commit/2018d20) **bugfix** RecordReader: properly return `io.ErrUnexpectedEOF`
- [#161](https://togithub.com/twmb/franz-go/pull/161) examples: avoid duplicated module name (thanks [@robsonpeixoto](https://togithub.com/robsonpeixoto))
### [`v1.5.0`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v150)
[Compare Source](https://togithub.com/twmb/franz-go/compare/v1.4.2...v1.5.0)
\===
This release adds a few new APIs, has a few small behavior changes, and has one
"breaking" change.
#### Breaking changes
The `kerberos` package is now a dedicated separate module. Rather than
requiring a major version bump, since this fix is entirely at the module level
for an almost entirely unused package, I figured it is *okayish* to technically
break compatibility for the few usages of this package, when the fix can be
done entirely when `go get`ing.
The [gokrb5](https://togithub.com/jcmturner/gokrb5) library, basically the only
library in the Go ecosystem that implements Kerberos, has a slightly [broken
license](https://togithub.com/jcmturner/gokrb5/issues/461). Organizations that
are sensitive to this were required to not use franz-go even if they did not
use Kerberos because franz-go pulls in a dependency on gokrb5.
Now, with `kerberos` being a distinct and separate module, depending on
franz-go only will *not* cause an indirect dependency on gokrb5.
If your upgrade is broken by this change, run:
```go
go get github.com/twmb/franz-go/pkg/sasl/kerberos@v1.0.0
go get github.com/twmb/franz-go@v1.5.0
```
#### Behavior changes
- `UnknownTopicRetries` now allows -1 to signal disabling the option (meaning
unlimited retries, rather than no retries). This follows the convention of
other options where -1 disables limits.
#### Improvements
- Waiting for unknown topics while producing now takes into account both the
produce context and aborting. Previously, the record context was only taken
into account *after* a topic was loaded. The same is true for aborting buffered
records: previously, abort would hang until a topic was loaded.
- New APIs are added to kmsg to deprecate the previous `Into` functions. The
`Into` functions still exist and will not be removed until kadm is stabilized
(see [#141](https://togithub.com/twmb/franz-go/issues/141)).
#### Features
- `ConsumeResetOffset` is now clearer, you can now use `NoResetOffset` with
start *or* end *or* exact offsets, and there is now the very useful
`Offset.AfterMilli` function. Previously, `NoResetOffset` only allowed starting
consuming at the start and it was not obvious why. We keep the previous
default-to-start behavior, but we now allow modifying it. As well, `AfterMilli`
can be used to largely replace `AtEnd`. Odds are, you want to consume all
records after your program starts *even if* new partitions are added to a
topic. Previously, if you added a partition to a topic, `AtEnd` would miss
records that were produced until the client refreshed metadata and discovered
the partition. Because of this, you were safer using `AtStart`, but this
unnecessarily forced you to consume everything on program start.
- Custom group balancers can now return errors, you can now intercept commits
to attach metadata, and you can now intercept offset fetches to read
metadata. Previously, none of this was possible. I considered metadata a bit
of a niche feature, but accessing it (as well as returning errors when
balancing) is required if you want to implement streams. New APIs now exist to
support the more advanced behavior: `PreCommitFnContext`, `OnOffsetsFetched`,
and `GroupMemberBalancerOrError`. As well, `BalancePlan.AsMemberIDMap` now
exists to provide access to a plan's underlying plan map. This did not exist
previously because I wanted to keep the type opaque for potential future
changes, but the odds of this are low and we can attempt forward compatibility
when the time arises.
- `RecordReader` now supports regular expressions for text values.
#### Relevant commits
- [`a2cbbf8`](https://togithub.com/twmb/franz-go/commit/a2cbbf8) go.{mod,sum}: go get -u ./...; go mod tidy
- [`ce7a84f`](https://togithub.com/twmb/franz-go/commit/ce7a84f) kerberos: split into dedicated module, p1
- [`e8e5c82`](https://togithub.com/twmb/franz-go/commit/e8e5c82) and [`744a60e`](https://togithub.com/twmb/franz-go/commit/744a60e) kgo: improve ConsumeResetOffset, NoResetOffset, add Offset.AfterMilli
- [`78fff0f`](https://togithub.com/twmb/franz-go/commit/78fff0f) and [`e8e5117`](https://togithub.com/twmb/franz-go/commit/e8e5117) and [`b457742`](https://togithub.com/twmb/franz-go/commit/b457742): add GroupMemberBalancerOrError
- [`b5256c7`](https://togithub.com/twmb/franz-go/commit/b5256c7) kadm: fix long standing poor API (Into fns)
- [`8148c55`](https://togithub.com/twmb/franz-go/commit/8148c55) BalancePlan: add AsMemberIDMap
- [`113a2c0`](https://togithub.com/twmb/franz-go/commit/113a2c0) add OnOffsetsFetched function to allow inspecting commit metadata
- [`0a4f2ec`](https://togithub.com/twmb/franz-go/commit/0a4f2ec) and [`cba9e26`](https://togithub.com/twmb/franz-go/commit/cba9e26) kgo: add PreCommitFnContext, enabling pre-commit interceptors for metadata
- [`42e5b57`](https://togithub.com/twmb/franz-go/commit/42e5b57) producer: allow a canceled context & aborting to quit unknown wait
- [`96d647a`](https://togithub.com/twmb/franz-go/commit/96d647a) UnknownTopicRetries: allow -1 to disable the option
- [`001c6d3`](https://togithub.com/twmb/franz-go/commit/001c6d3) RecordReader: support regular expressions for text values
### [`v1.4.2`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v142)
[Compare Source](https://togithub.com/twmb/franz-go/compare/v1.4.1...v1.4.2)
\===
This release fixes a potential incremental fetch session spin loop /
undesirable behavior. This was not reported, but can happen if you use many
clients against your cluster.
Previously, if a broker advertised that it supported consumer fetch sessions
but did not actually create any and returned "0" to signify no session was
created, the client would accept that 0 as a new fetch session. If the fetch
response returned no data and thus made no forward progress, the next fetch
request would include no partitions (believing a fetch session was created),
and the broker would again reply immediately with no data and no fetch session.
This would loop. Now, if the broker indicates no fetch session was created, we
immediately stop trying to create new fetch sessions and never try again.
In practice, fetch sessions are rejected if and replied to with 0 if a new one
cannot be created. The default fetch session cache in Kafka is 1,000. If you
have more than 1,000 active clients (where brokers count as clients against
other brokers), you are at risk of this bug.
This bug would manifest in clearly visible ways: higher cpu, no forward
progress while consuming. If you have not seen these, you have not experienced
the bug. However, it is recommended that all users upgrade to avoid it.
This has two followup fixes to [`83b0a32`][83b0a32], one which fixes behavior
that broke `EndBeginTxnSafe`, and one which mirrors some of the logic
supporting `EndBeginTxnUnsafe` into `EndTransaction` itself. This also fixes a
very rare data race that *realistically* would result in a new connection being
killed immediately (since at the CPU, reads/writes of pointers is atomic).
- [`2faf459`](https://togithub.com/twmb/franz-go/commit/2faf459) **bugfix** broker: fix rare data race
- [`8f7c8cd`](https://togithub.com/twmb/franz-go/commit/8f7c8cd) **bugfix** EndBeginTxnUnsafe: partially back out of [`83b0a32`][83b0a32]
- [`85a680e`](https://togithub.com/twmb/franz-go/commit/85a680e) **bugfix** consuming: do not continually try to create fetch sessions
- [`2decd27`](https://togithub.com/twmb/franz-go/commit/2decd27) **bugfix** EndTransaction: mirror EndBeginTxnUnsafe logic
[83b0a32]: https://togithub.com/twmb/franz-go/commit/83b0a32
### [`v1.4.1`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v141)
[Compare Source](https://togithub.com/twmb/franz-go/compare/v1.4.0...v1.4.1)
\===
This release pins kmsg to its newly stable version v1.0.0, fixing a compilation
issue for anybody doing `go get -u`.
The kmsg package was previously unversioned because Kafka sometimes changes the
protocol in such a way that breaks the API as chosen in kmsg (plain types, not
objects for everything). The most recent change in kmsg to match a recent type
rename in Kafka broke kgo because kgo depended on the old name. Again, this was
not pinned because franz-go did not depend on a specific version of kmsg. To
prevent this issue from happening again, we now pin to a stable kmsg version.
There are also two small bugfixes and a few improvements. Previously,
`ProducerFenced` was marked as retriable, which could result in the client
internally entering a fatal state that the user was unaware of. This should now
be bubbled up. As well, there were a few subtle issues with `EndBeginTxnUnsafe`
that have been addressed.
Notable commits & PRs:
- [`83b0a32`](https://togithub.com/twmb/franz-go/commit/83b0a32) **bugfix** kgo: EndAndBeginTransaction w/ EndBeginTxnUnsafe: fix three issues
- [`bd1d43d`](https://togithub.com/twmb/franz-go/commit/bd1d43d) sink: small AddPartitionsToTxn improvements
- [`65ca0bd`](https://togithub.com/twmb/franz-go/commit/65ca0bd) **bugfix** kerr: ProducerFenced is not retriable
- [PR #148](https://togithub.com/twmb/franz-go/pull/148) lower `FetchMaxPartitionBytes` to 1MiB to be in line with the Kafka default (thanks [@jcsp](https://togithub.com/jcsp))
- [`806cf53`](https://togithub.com/twmb/franz-go/commit/806cf53) **feature** kmsg: add TextMarshaler/TextUnmarshaler to enums
- [`49f678d`](https://togithub.com/twmb/franz-go/commit/49f678d) update deps, pulling in klauspost/compress v1.15.1 which makes zstd encoding & decoding stateless
### [`v1.4.0`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v140)
[Compare Source](https://togithub.com/twmb/franz-go/compare/v1.3.5...v1.4.0)
\===
This release adds a lot of new features and changes a few internal behaviors.
The new features have been tested, but it is possible that a bug slipped by—if
you see one, please open an issue and the bug can be fixed promptly.
#### Behavior changes
- **Promises are now serialized**. Previously, promises were called at the end
of handling produce requests. As well, errors that caused records to fail
independent of producing could fail whenever. Now, all promises are called in
one loop. Benchmarking showed that concurrent promises did not really help,
even in cases where the promises could be concurrent. As well, my guess is that
most people serialize promises, resulting in more complicated logic punted to
the users. Now with serializing promises, user code can be simpler.
- The default `MetadataMinAge` has been lowered from 5s to 2.5s. Metadata
refreshes internally on retryable errors, 2.5s helps fail records for
non-existing topics quicker. Related, for sharded requests, we now cache topic
& partition metadata for the `MetadataMinAge`. This mostly benefits
`ListOffsets`, where usually a person may list both the start and end back to
back. We cannot cache indefinitely because a user may add partitions outside
this client, but 2.5s is still helpful especially for how infrequently sharded
requests are issued.
- Group leaders now track topics that the leader is not interested in
consuming. Previously, if leader A consumed only topic foo and member B only
bar, then leader A would not notice if partitions were added to bar. Now, the
leader tracks bar. This behavior change only affects groups where the members
consume non-overlapping topics.
- Group joins & leaves now include a reason, as per KIP-800. This will be
useful when Kafka 3.2 is released.
- Transactions no longer log `CONCURRENT_TRANSACTIONS` errors at the info
level. This was a noisy log that meant nothing and was non-actionable. We
still track this at the debug level.
#### Features
A few new APIs and options have been added. These will be described shortly
here, and the commits are linked below.
- `ConcurrentTransactionsBackoff`: a new option that allows configuring the
backoff when starting a transaction runs into the `CONCURRENT_TRANSACTIONS`
error. Changing the backoff can decrease latency if Kafka is fast, but can
increase load on the cluster.
- `MaxProduceRequestsInflightPerBroker`: a new option that allows changing the
max inflight produce requests per broker *if* you disable idempotency.
Idempotency has an upper bound of 5 requests; by default, disabling idempotency
sets the max inflight to 1.
- `UnknownTopicRetries`: a new option that sets how many times a metadata load
for a topic can return `UNKNOWN_TOPIC_OR_PARTITION` before all records
buffered for the topic are failed. As well, we now use this option more widely:
if a topic is loaded successfully and then later repeatedly experiences these
errors, records will be failed. Previously, this limit was internal and was
only applied before the topic was loaded successfully once.
- `NoResetOffset`: a new special offset that can be used with
`ConsumeResetOffset` to trigger the client to enter a fatal state if
`OffsetOutOfRange` is encountered.
- `Client.PurgeTopicsFromClient`: a new API that allows for completely removing
a topic from the client. This can help if you regex consume and delete a topic,
or if you produce to random topics and then stop producing to some of them.
- `Client.AddConsumeTopics`: a new API that enables you to consume from topics
that you did not initially configure. This enables you to add more topics to
consume from without restarting the client; this works both both direct
consumers and group consumers.
- `Client.TryProduce`: a new API that is a truly non-blocking produce. If the
client has the maximum amount of records buffered, this function will
immediately fail a new promise with `ErrMaxBuffered`.
- `Client.ForceMetadataRefresh`: a new API that allows you to manually trigger
a metadata refresh. This can be useful if you added partitions to a topic and
want to trigger a metadata refresh to load those partitions sooner than the
default `MetadataMaxAge` refresh interval.
- `Client.EndAndBeginTransaction`: a new API that can be used to have higher
throughput when producing transactionally. This API requires care; if you use
it, read the documentation for what it provides and any downsides.
- `BlockRebalancesOnPoll` and `Client.AllowRebalance`: a new option and
corresponding required API that allows for easier reasoning about when
rebalances can happen. This option can be greatly beneficial to users for
simplifying code, but has a risk around taking so long that your group member
is booted from the group. Two examples were added using these options.
- `kversion.V3_1_0`: the kversion package now officially detects v3.1 and has
an API for it.
#### Relevant commits
- [PR #137](https://togithub.com/twmb/franz-go/pull/137) and [`c3fc8e0`](https://togithub.com/twmb/franz-go/commit/c3fc8e0): add two more goroutine per consumer examples (thanks [@JacobSMoller](https://togithub.com/JacobSMoller))
- [`cffbee7`](https://togithub.com/twmb/franz-go/commit/cffbee7) consumer: add BlockRebalancesOnPoll option, AllowRebalance (commit accidentally pluralized)
- [`39af436`](https://togithub.com/twmb/franz-go/commit/39af436) docs: add metrics-and-logging.md
- [`83dfa9d`](https://togithub.com/twmb/franz-go/commit/83dfa9d) client: add EndAndBeginTransaction
- [`d11066f`](https://togithub.com/twmb/franz-go/commit/d11066f) committing: internally retry on some errors when cooperative
- [`31f3f5f`](https://togithub.com/twmb/franz-go/commit/31f3f5f) producer: serialize promises
- [`e3ef142`](https://togithub.com/twmb/franz-go/commit/e3ef142) txn: move concurrent transactions log to debug level
- [`10ee8dd`](https://togithub.com/twmb/franz-go/commit/10ee8dd) group consuming: add reasons to JoinGroup, LeaveGroup per KIP-800
- [`0bfaf64`](https://togithub.com/twmb/franz-go/commit/0bfaf64) consumer group: track topics that the leader is not interested in
- [`e8495bb`](https://togithub.com/twmb/franz-go/commit/e8495bb) client: add ForceMetadataRefresh
- [`c763c9b`](https://togithub.com/twmb/franz-go/commit/c763c9b) consuming: add NoResetOffset
- [`4e0e1d7`](https://togithub.com/twmb/franz-go/commit/4e0e1d7) config: add UnknownTopicRetries option, use more widely
- [`7f58a97`](https://togithub.com/twmb/franz-go/commit/7f58a97) config: lower default MetadataMinAge to 2.5s
- [`e7bd28f`](https://togithub.com/twmb/franz-go/commit/e7bd28f) Client,GroupTransactSession: add TryProduce
- [`2a2cf66`](https://togithub.com/twmb/franz-go/commit/2a2cf66) consumer: add AddConsumeTopics
- [`d178e26`](https://togithub.com/twmb/franz-go/commit/d178e26) client: add PurgeTopicsFromClient
- [`336d2c9`](https://togithub.com/twmb/franz-go/commit/336d2c9) kgo: add ConcurrentTransactionsBackoff, MaxProduceRequestsInflightPerBroker
- [`fb04711`](https://togithub.com/twmb/franz-go/commit/fb04711) kversion: cut v3.1
### [`v1.3.5`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v135)
[Compare Source](https://togithub.com/twmb/franz-go/compare/v1.3.4...v1.3.5)
\===
This patch release fixes a panic in `GroupTransactSession.End` and has three
behavior changes that are beneficial to users. The panic was introduced in
v1.3.0; if using a `GroupTransactSession`, it is recommended you upgrade to
this release.
The next release aims to be v1.4.0, this release is a small one to address a
few issues before the much larger and feature filled v1.4 release.
- [`010e8e1`](https://togithub.com/twmb/franz-go/commit/010e8e1) txn: fix panic in GroupTransactSession.End
- [`f9cd625`](https://togithub.com/twmb/franz-go/commit/f9cd625) consuming: handle exact offset consuming better
- [`2ab1978`](https://togithub.com/twmb/franz-go/commit/2ab1978) EndTransaction: return nil rather than an error if not in a transaction
- [`96bfe52`](https://togithub.com/twmb/franz-go/commit/96bfe52) broker: remove 5s minimum for sasl session lifetime
### [`v1.3.4`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v134)
[Compare Source](https://togithub.com/twmb/franz-go/compare/v1.3.3...v1.3.4)
\===
This small patch release fixes a problem with
[`4f2e7fe3`](https://togithub.com/twmb/franz-go/commit/4f2e7fe3) which was meant
to address [#98](https://togithub.com/twmb/franz-go/issues/98). The fix was not
complete in that the fix would only trigger if a group member had partitions
added to it. We now rearrange the logic such that it occurs always. This bug
was found while making a change in the code in support of a new feature in the
v1.4 branch; this bug was not encountered in production.
This also bumps the franz-go/pkg/kmsg dependency so that `JoinGroup.Reason` is
properly tagged as v9+.
The next release will be v1.4, which is nearly ready to be merged into this
branch and tagged. Follow issue [#135](https://togithub.com/twmb/franz-go/issues/135) for more details.
- [`02560c7`](https://togithub.com/twmb/franz-go/commit/02560c7) consumer group: bugfix fetch offsets spanning rebalance
### [`v1.3.3`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v133)
[Compare Source](https://togithub.com/twmb/franz-go/compare/v1.3.2...v1.3.3)
\===
This patch release contains two minor bug fixes and a few small behavior
changes. The upcoming v1.4 release will contain more changes to lint the entire
codebase and will have a few new options as features to configure the client
internals. There are a few features in development yet that I would like to
complete before tagging the next minor release.
#### Bug fixes
Repeated integration testing resulted in a rare data race, and one other bug
was found by linting. For the race, if a group was left *or* heartbeating
stopped *before* offset fetching finished, then there would be a concurrent
double write to an error variable: one write would try to write an error from
the request being cut (which would be `context.Canceled`), and the other write
would write the same error, but directly from `ctx.Err`. Since both of these
are the same type pointer and data pointer, it is unlikely this race would
result in anything if it was ever encountered, and encountering it would be
rare.
For the second bug, after this prior one, I wondered if any linter would have
caught this bug (the answer is no). However, in the process of heavily linting
the code base, a separate bug was found. This bug has **no impact**, but it is
good to fix. If you previously consumed and specified *exact* offsets to
consume from, the internal `ListOffsets` request would use that offset as a
timestamp, rather than using -1 as I meant to internally. The whole point is
just to load the partition, so using a random number for a timestmap is just as
good as using -1, but we may as well use -1 to be proper.
#### Behavior changes
Previously when producing, the buffer draining goroutine would sleep for 50ms
whenever it started. This was done to give high throughput producers a chance
to produce more records within the first batch, rather than the loop being so
fast that one record is in the first batch (and more in the others). This 50ms
sleep is a huge penalty to oneshot producers (producing one message at a time,
synchronously) and also was a slight penalty to high throughput producers
whenever the drain loop quit and needed to be resumed. We now eliminate this
sleep. This may result in more smaller batches, but definitely helps oneshot
producers. A linger can be used to avoid small batches, if needed.
Previously, due to MS Azure improperly replying to produce requests when acks
were zero, a discard goroutine was added to drain the produce connection if the
client was configured to produce with no acks. Logic has been added to quit
this goroutine if nothing is read on that goroutine for 3x the connection
timeout overhead, which is well enough time that a response should be received
if the broker is ever going to send one.
Previously, `MarkCommitRecords` would forbid rewinds and only allow advancing
offsets. The code now allows rewinds if you mark an early offset after you have
already marked a later offset. This brings the behavior in line with the
current `CommitOffsets`.
Previously, if the client encountered `CONCURRENT_TRANSACTIONS` during transactions,
it would sleep for 100ms and then retry the relevant request. This sleep has been
dropped to 20ms, which should help latency when transacting quickly. The v1.4
release will allow this number to be configured with a new option.
#### Additions
KIP-784 and KIP-814 are now supported (unreleased yet in Kafka). Support for
KIP-814 required bumping the franz-go's kmsg dep. Internally, only KIP-814
affects client behavior, but since this is unrelased, it is not changing any
behavior.
#### Relevant commits
- [`b39ca31`](https://togithub.com/twmb/franz-go/commit/b39ca31) fetchOffsets: fix data race
- [`4156e9f`](https://togithub.com/twmb/franz-go/commit/4156e9f) kgo: fix one bug found by linting
- [`72760bf..ad991d8`](https://togithub.com/twmb/franz-go/compare/72760bf..ad991d8) kmsg, kversion, kgo: support KIP-814 (SkipAssignment in JoinGroupResponse)
- [`db9017a`](https://togithub.com/twmb/franz-go/commit/db9017a) broker: allow the acks==0 producing discard goroutine to die
- [`eefb1f3`](https://togithub.com/twmb/franz-go/commit/eefb1f3) consuming: update docs & simplify; `MarkCommitRecords`: allow rewinds
- [`8808b94`](https://togithub.com/twmb/franz-go/commit/8808b94) sink: remove 50ms wait on new drain loops
- [`a13f918`](https://togithub.com/twmb/franz-go/commit/a13f918) kmsg & kversion: add support for KIP-784 (ErrorCode in DescribeLogDirs response)
- [PR #133](https://togithub.com/twmb/franz-go/pull/133) - lower concurrent transactions retry to 20ms; configurability will be in the next release (thanks [@eduard-netsajev](https://togithub.com/eduard-netsajev))
### [`v1.3.2`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v132)
[Compare Source](https://togithub.com/twmb/franz-go/compare/v1.3.1...v1.3.2)
\===
This patch fixes a bug of unclear severity related to transactions. Credit goes
to [@eduard-netsajev](https://togithub.com/eduard-netsajev) for finding this long
standing problem.
In Kafka, if you try to start a transaction too soon after finishing the
previous one, Kafka may not actually have internally finished the prior
transaction yet and can return a `CONCURRENT_TRANSACTIONS` error. To work
around this, clients are expected to retry when they see this error (even
though it is marked as not retriable).
This client does that properly, but unfortunately did not bubble up any *non*
`CONCURRENT_TRANSACTIONS` errors.
From the code, it *appears* as if in the worst case, this could have meant that
transactions invisibly looked like they were working and being used when they
actually were not. However, it's likely that other errors would be noticed
internally, and it's possible that if you encountered problems, the entire ETL
pipeline would stall anyway.
All told, it's not entirely clear what the ramifications for this bug are, and
it is recommended that if you use transactions, you should update immediately.
- [PR #131](https://togithub.com/twmb/franz-go/pull/131) - txns: don't ignore error in doWithConcurrentTransactions
### [`v1.3.1`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v131)
[Compare Source](https://togithub.com/twmb/franz-go/compare/v1.3.0...v1.3.1)
\===
This small patch release fixes a leaked-goroutine problem after closing a
consuming client, and adds one config validation to hopefully reduce confusion.
For the bug, if a fetch was buffered when you were closing the client, an
internal goroutine would stay alive. In normal cases where you have one client
in your program and you close it on program shutdown, this leak is not really
important. If your program recreates consuming clients often and stays alive,
the leaked goroutine could eventually result in unexpected memory consumption.
Now, the client internally unbuffers all fetches at the end of `Close`, which
allows the previously-leaking goroutine to exit.
For the config validation, using `ConsumePartitions` and `ConsumeTopics` with
the same topic in both options would silently ignore the topic in
`ConsumePartitions`. Now, the client will return an error that using the same
topic in both options is invalid.
- [`ea11266`](https://togithub.com/twmb/franz-go/commit/ea11266) config: add duplicately specified topic validation
- [`bb581f4`](https://togithub.com/twmb/franz-go/commit/bb581f4) **bugfix** client: unbuffer fetches on Close to allow mangeFetchConcurrency to quit
### [`v1.3.0`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v130)
[Compare Source](https://togithub.com/twmb/franz-go/compare/v1.2.6...v1.3.0)
\===
This release contains three new features, a few behavior changes, and one minor
bugfix.
For features, you can now adjust fetched offsets before they are used (thanks
[@michaelwilner](https://togithub.com/michaelwilner)!), you can now "ping" your cluster to see if the client can
connect at all, and you can now use `SetOffsets` when consuming partitions
manually. As a somewhat of a feature-ish, producing no longer requires a
context, instead if a context is nil, `context.Background` is used (this was
added to allow more laziness when writing small unimportant files).
The transactional behavior change is important: the documentation changes are
worth reading, and it is worth using a 2.5+ cluster along with the
`RequireStableFetchOffsets` option if possible. The metadata leader epoch
rewinding behavior change allows the client to continue in the event of odd
cluster issues.
In kadm, we now return individual per-partition errors if partitions are not
included in OffsetCommit responses. The generated code now has a few more enums
(thanks [@weeco](https://togithub.com/weeco)!)
Lastly, as a small bugfix, `client.Close()` did not properly stop seed brokers.
A previous commit split seed brokers and non-seed brokers internally into two
fields but did not add broker shutdown on the now-split seed broker field.
- [`e0b520c`](https://togithub.com/twmb/franz-go/commit/e0b520c) **behavior change** kadm: set per-partition errors on missing offsets in CommitOffsets
- [`32425df`](https://togithub.com/twmb/franz-go/commit/32425df) **feature** client: add Ping method
- [`a059901`](https://togithub.com/twmb/franz-go/commit/a059901) **behavior change** txns: sleep 200ms on commit, preventing rebalance / new commit
- [`12eaa1e`](https://togithub.com/twmb/franz-go/commit/12eaa1e) **behavior change** metadata: allow leader epoch rewinds after 5 tries
- [`029e655`](https://togithub.com/twmb/franz-go/commit/029e655) **feature-ish** Produce{,Sync}: default to context.Background if no ctx is provided
- [`eb2cec3`](https://togithub.com/twmb/franz-go/commit/eb2cec3) **bugfix** client: stop seed brokers on client.Close
- [`2eae20d`](https://togithub.com/twmb/franz-go/commit/2eae20d) **feature** consumer: allow SetOffsets for direct partition consuming
- [pr #120](https://togithub.com/twmb/franz-go/pull/120) **feature** Add groupopt to swizzle offset assignments before consumption (thanks [@michaelwilner](https://togithub.com/michaelwilner)!)
Configuration
📅 Schedule: At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
[ ] If you want to rebase/retry this PR, click this checkbox.
This PR contains the following updates:
v1.2.6->v1.5.2Release Notes
twmb/franz-go
### [`v1.5.2`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v152) [Compare Source](https://togithub.com/twmb/franz-go/compare/v1.5.1...v1.5.2) \=== This tiny release contains another fix for `RecordReader`, specifically for regex parsing, and changes the behavior of `MarkCommitRecords` to its original behavior. - For `RecordReader`, the regex type was not properly registered everywhere internally. This resulted in `%v{re#...#}\n` trying to parse a newline as part of the regex, rather than as a delimiter. This is fixed and a test case added. Note that this feature is niche *and* new, so this is a very minor bug. - For `MarkCommitRecords`, v1.3.3 changed the behavior to allow rewinds. This is difficult to reason about and can result in bugs. More likely, you just want to periodically mark to move forward; the behavior now is more in line with what people expect. - [`ff5a3ed`](https://togithub.com/twmb/franz-go/commit/ff5a3ed) MarkCommitRecords: forbid rewinds - [`41284b3`](https://togithub.com/twmb/franz-go/commit/41284b3) RecordReader: fix regex reading even more ### [`v1.5.1`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v151) [Compare Source](https://togithub.com/twmb/franz-go/compare/v1.5.0...v1.5.1) \=== This release fixes a minor bug in `RecordReader`, and has a behavior change for polling records (and a corresponding tiny new helper API for this). For the bugfix, `RecordReader` did not always properly return `io.ErrUnexpectedEOF`. We now return it more properly and add tests for the missing edge cases. For the behavior change, we now inject an error into `PollFetches` and `PollRecords` if the user context is canceled. Previously, we would just quit the poll and return. This change introduces a new corresponding function, `Fetches.Err() error`. The thought here is that now we have more injected errors, so `Fetches.Err` can make it easier to check various errors. - [`cbc8962`](https://togithub.com/twmb/franz-go/commit/cbc8962) **behavior change** Poll{Records,Fetches}: inject an error for canceled context - [#163](https://togithub.com/twmb/franz-go/pull/163) docs: updates StickyBalancer godoc (thanks [@Streppel](https://togithub.com/Streppel)) - [`2018d20`](https://togithub.com/twmb/franz-go/commit/2018d20) **bugfix** RecordReader: properly return `io.ErrUnexpectedEOF` - [#161](https://togithub.com/twmb/franz-go/pull/161) examples: avoid duplicated module name (thanks [@robsonpeixoto](https://togithub.com/robsonpeixoto)) ### [`v1.5.0`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v150) [Compare Source](https://togithub.com/twmb/franz-go/compare/v1.4.2...v1.5.0) \=== This release adds a few new APIs, has a few small behavior changes, and has one "breaking" change. #### Breaking changes The `kerberos` package is now a dedicated separate module. Rather than requiring a major version bump, since this fix is entirely at the module level for an almost entirely unused package, I figured it is *okayish* to technically break compatibility for the few usages of this package, when the fix can be done entirely when `go get`ing. The [gokrb5](https://togithub.com/jcmturner/gokrb5) library, basically the only library in the Go ecosystem that implements Kerberos, has a slightly [broken license](https://togithub.com/jcmturner/gokrb5/issues/461). Organizations that are sensitive to this were required to not use franz-go even if they did not use Kerberos because franz-go pulls in a dependency on gokrb5. Now, with `kerberos` being a distinct and separate module, depending on franz-go only will *not* cause an indirect dependency on gokrb5. If your upgrade is broken by this change, run: ```go go get github.com/twmb/franz-go/pkg/sasl/kerberos@v1.0.0 go get github.com/twmb/franz-go@v1.5.0 ``` #### Behavior changes - `UnknownTopicRetries` now allows -1 to signal disabling the option (meaning unlimited retries, rather than no retries). This follows the convention of other options where -1 disables limits. #### Improvements - Waiting for unknown topics while producing now takes into account both the produce context and aborting. Previously, the record context was only taken into account *after* a topic was loaded. The same is true for aborting buffered records: previously, abort would hang until a topic was loaded. - New APIs are added to kmsg to deprecate the previous `Into` functions. The `Into` functions still exist and will not be removed until kadm is stabilized (see [#141](https://togithub.com/twmb/franz-go/issues/141)). #### Features - `ConsumeResetOffset` is now clearer, you can now use `NoResetOffset` with start *or* end *or* exact offsets, and there is now the very useful `Offset.AfterMilli` function. Previously, `NoResetOffset` only allowed starting consuming at the start and it was not obvious why. We keep the previous default-to-start behavior, but we now allow modifying it. As well, `AfterMilli` can be used to largely replace `AtEnd`. Odds are, you want to consume all records after your program starts *even if* new partitions are added to a topic. Previously, if you added a partition to a topic, `AtEnd` would miss records that were produced until the client refreshed metadata and discovered the partition. Because of this, you were safer using `AtStart`, but this unnecessarily forced you to consume everything on program start. - Custom group balancers can now return errors, you can now intercept commits to attach metadata, and you can now intercept offset fetches to read metadata. Previously, none of this was possible. I considered metadata a bit of a niche feature, but accessing it (as well as returning errors when balancing) is required if you want to implement streams. New APIs now exist to support the more advanced behavior: `PreCommitFnContext`, `OnOffsetsFetched`, and `GroupMemberBalancerOrError`. As well, `BalancePlan.AsMemberIDMap` now exists to provide access to a plan's underlying plan map. This did not exist previously because I wanted to keep the type opaque for potential future changes, but the odds of this are low and we can attempt forward compatibility when the time arises. - `RecordReader` now supports regular expressions for text values. #### Relevant commits - [`a2cbbf8`](https://togithub.com/twmb/franz-go/commit/a2cbbf8) go.{mod,sum}: go get -u ./...; go mod tidy - [`ce7a84f`](https://togithub.com/twmb/franz-go/commit/ce7a84f) kerberos: split into dedicated module, p1 - [`e8e5c82`](https://togithub.com/twmb/franz-go/commit/e8e5c82) and [`744a60e`](https://togithub.com/twmb/franz-go/commit/744a60e) kgo: improve ConsumeResetOffset, NoResetOffset, add Offset.AfterMilli - [`78fff0f`](https://togithub.com/twmb/franz-go/commit/78fff0f) and [`e8e5117`](https://togithub.com/twmb/franz-go/commit/e8e5117) and [`b457742`](https://togithub.com/twmb/franz-go/commit/b457742): add GroupMemberBalancerOrError - [`b5256c7`](https://togithub.com/twmb/franz-go/commit/b5256c7) kadm: fix long standing poor API (Into fns) - [`8148c55`](https://togithub.com/twmb/franz-go/commit/8148c55) BalancePlan: add AsMemberIDMap - [`113a2c0`](https://togithub.com/twmb/franz-go/commit/113a2c0) add OnOffsetsFetched function to allow inspecting commit metadata - [`0a4f2ec`](https://togithub.com/twmb/franz-go/commit/0a4f2ec) and [`cba9e26`](https://togithub.com/twmb/franz-go/commit/cba9e26) kgo: add PreCommitFnContext, enabling pre-commit interceptors for metadata - [`42e5b57`](https://togithub.com/twmb/franz-go/commit/42e5b57) producer: allow a canceled context & aborting to quit unknown wait - [`96d647a`](https://togithub.com/twmb/franz-go/commit/96d647a) UnknownTopicRetries: allow -1 to disable the option - [`001c6d3`](https://togithub.com/twmb/franz-go/commit/001c6d3) RecordReader: support regular expressions for text values ### [`v1.4.2`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v142) [Compare Source](https://togithub.com/twmb/franz-go/compare/v1.4.1...v1.4.2) \=== This release fixes a potential incremental fetch session spin loop / undesirable behavior. This was not reported, but can happen if you use many clients against your cluster. Previously, if a broker advertised that it supported consumer fetch sessions but did not actually create any and returned "0" to signify no session was created, the client would accept that 0 as a new fetch session. If the fetch response returned no data and thus made no forward progress, the next fetch request would include no partitions (believing a fetch session was created), and the broker would again reply immediately with no data and no fetch session. This would loop. Now, if the broker indicates no fetch session was created, we immediately stop trying to create new fetch sessions and never try again. In practice, fetch sessions are rejected if and replied to with 0 if a new one cannot be created. The default fetch session cache in Kafka is 1,000. If you have more than 1,000 active clients (where brokers count as clients against other brokers), you are at risk of this bug. This bug would manifest in clearly visible ways: higher cpu, no forward progress while consuming. If you have not seen these, you have not experienced the bug. However, it is recommended that all users upgrade to avoid it. This has two followup fixes to [`83b0a32`][83b0a32], one which fixes behavior that broke `EndBeginTxnSafe`, and one which mirrors some of the logic supporting `EndBeginTxnUnsafe` into `EndTransaction` itself. This also fixes a very rare data race that *realistically* would result in a new connection being killed immediately (since at the CPU, reads/writes of pointers is atomic). - [`2faf459`](https://togithub.com/twmb/franz-go/commit/2faf459) **bugfix** broker: fix rare data race - [`8f7c8cd`](https://togithub.com/twmb/franz-go/commit/8f7c8cd) **bugfix** EndBeginTxnUnsafe: partially back out of [`83b0a32`][83b0a32] - [`85a680e`](https://togithub.com/twmb/franz-go/commit/85a680e) **bugfix** consuming: do not continually try to create fetch sessions - [`2decd27`](https://togithub.com/twmb/franz-go/commit/2decd27) **bugfix** EndTransaction: mirror EndBeginTxnUnsafe logic [83b0a32]: https://togithub.com/twmb/franz-go/commit/83b0a32 ### [`v1.4.1`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v141) [Compare Source](https://togithub.com/twmb/franz-go/compare/v1.4.0...v1.4.1) \=== This release pins kmsg to its newly stable version v1.0.0, fixing a compilation issue for anybody doing `go get -u`. The kmsg package was previously unversioned because Kafka sometimes changes the protocol in such a way that breaks the API as chosen in kmsg (plain types, not objects for everything). The most recent change in kmsg to match a recent type rename in Kafka broke kgo because kgo depended on the old name. Again, this was not pinned because franz-go did not depend on a specific version of kmsg. To prevent this issue from happening again, we now pin to a stable kmsg version. There are also two small bugfixes and a few improvements. Previously, `ProducerFenced` was marked as retriable, which could result in the client internally entering a fatal state that the user was unaware of. This should now be bubbled up. As well, there were a few subtle issues with `EndBeginTxnUnsafe` that have been addressed. Notable commits & PRs: - [`83b0a32`](https://togithub.com/twmb/franz-go/commit/83b0a32) **bugfix** kgo: EndAndBeginTransaction w/ EndBeginTxnUnsafe: fix three issues - [`bd1d43d`](https://togithub.com/twmb/franz-go/commit/bd1d43d) sink: small AddPartitionsToTxn improvements - [`65ca0bd`](https://togithub.com/twmb/franz-go/commit/65ca0bd) **bugfix** kerr: ProducerFenced is not retriable - [PR #148](https://togithub.com/twmb/franz-go/pull/148) lower `FetchMaxPartitionBytes` to 1MiB to be in line with the Kafka default (thanks [@jcsp](https://togithub.com/jcsp)) - [`806cf53`](https://togithub.com/twmb/franz-go/commit/806cf53) **feature** kmsg: add TextMarshaler/TextUnmarshaler to enums - [`49f678d`](https://togithub.com/twmb/franz-go/commit/49f678d) update deps, pulling in klauspost/compress v1.15.1 which makes zstd encoding & decoding stateless ### [`v1.4.0`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v140) [Compare Source](https://togithub.com/twmb/franz-go/compare/v1.3.5...v1.4.0) \=== This release adds a lot of new features and changes a few internal behaviors. The new features have been tested, but it is possible that a bug slipped by—if you see one, please open an issue and the bug can be fixed promptly. #### Behavior changes - **Promises are now serialized**. Previously, promises were called at the end of handling produce requests. As well, errors that caused records to fail independent of producing could fail whenever. Now, all promises are called in one loop. Benchmarking showed that concurrent promises did not really help, even in cases where the promises could be concurrent. As well, my guess is that most people serialize promises, resulting in more complicated logic punted to the users. Now with serializing promises, user code can be simpler. - The default `MetadataMinAge` has been lowered from 5s to 2.5s. Metadata refreshes internally on retryable errors, 2.5s helps fail records for non-existing topics quicker. Related, for sharded requests, we now cache topic & partition metadata for the `MetadataMinAge`. This mostly benefits `ListOffsets`, where usually a person may list both the start and end back to back. We cannot cache indefinitely because a user may add partitions outside this client, but 2.5s is still helpful especially for how infrequently sharded requests are issued. - Group leaders now track topics that the leader is not interested in consuming. Previously, if leader A consumed only topic foo and member B only bar, then leader A would not notice if partitions were added to bar. Now, the leader tracks bar. This behavior change only affects groups where the members consume non-overlapping topics. - Group joins & leaves now include a reason, as per KIP-800. This will be useful when Kafka 3.2 is released. - Transactions no longer log `CONCURRENT_TRANSACTIONS` errors at the info level. This was a noisy log that meant nothing and was non-actionable. We still track this at the debug level. #### Features A few new APIs and options have been added. These will be described shortly here, and the commits are linked below. - `ConcurrentTransactionsBackoff`: a new option that allows configuring the backoff when starting a transaction runs into the `CONCURRENT_TRANSACTIONS` error. Changing the backoff can decrease latency if Kafka is fast, but can increase load on the cluster. - `MaxProduceRequestsInflightPerBroker`: a new option that allows changing the max inflight produce requests per broker *if* you disable idempotency. Idempotency has an upper bound of 5 requests; by default, disabling idempotency sets the max inflight to 1. - `UnknownTopicRetries`: a new option that sets how many times a metadata load for a topic can return `UNKNOWN_TOPIC_OR_PARTITION` before all records buffered for the topic are failed. As well, we now use this option more widely: if a topic is loaded successfully and then later repeatedly experiences these errors, records will be failed. Previously, this limit was internal and was only applied before the topic was loaded successfully once. - `NoResetOffset`: a new special offset that can be used with `ConsumeResetOffset` to trigger the client to enter a fatal state if `OffsetOutOfRange` is encountered. - `Client.PurgeTopicsFromClient`: a new API that allows for completely removing a topic from the client. This can help if you regex consume and delete a topic, or if you produce to random topics and then stop producing to some of them. - `Client.AddConsumeTopics`: a new API that enables you to consume from topics that you did not initially configure. This enables you to add more topics to consume from without restarting the client; this works both both direct consumers and group consumers. - `Client.TryProduce`: a new API that is a truly non-blocking produce. If the client has the maximum amount of records buffered, this function will immediately fail a new promise with `ErrMaxBuffered`. - `Client.ForceMetadataRefresh`: a new API that allows you to manually trigger a metadata refresh. This can be useful if you added partitions to a topic and want to trigger a metadata refresh to load those partitions sooner than the default `MetadataMaxAge` refresh interval. - `Client.EndAndBeginTransaction`: a new API that can be used to have higher throughput when producing transactionally. This API requires care; if you use it, read the documentation for what it provides and any downsides. - `BlockRebalancesOnPoll` and `Client.AllowRebalance`: a new option and corresponding required API that allows for easier reasoning about when rebalances can happen. This option can be greatly beneficial to users for simplifying code, but has a risk around taking so long that your group member is booted from the group. Two examples were added using these options. - `kversion.V3_1_0`: the kversion package now officially detects v3.1 and has an API for it. #### Relevant commits - [PR #137](https://togithub.com/twmb/franz-go/pull/137) and [`c3fc8e0`](https://togithub.com/twmb/franz-go/commit/c3fc8e0): add two more goroutine per consumer examples (thanks [@JacobSMoller](https://togithub.com/JacobSMoller)) - [`cffbee7`](https://togithub.com/twmb/franz-go/commit/cffbee7) consumer: add BlockRebalancesOnPoll option, AllowRebalance (commit accidentally pluralized) - [`39af436`](https://togithub.com/twmb/franz-go/commit/39af436) docs: add metrics-and-logging.md - [`83dfa9d`](https://togithub.com/twmb/franz-go/commit/83dfa9d) client: add EndAndBeginTransaction - [`d11066f`](https://togithub.com/twmb/franz-go/commit/d11066f) committing: internally retry on some errors when cooperative - [`31f3f5f`](https://togithub.com/twmb/franz-go/commit/31f3f5f) producer: serialize promises - [`e3ef142`](https://togithub.com/twmb/franz-go/commit/e3ef142) txn: move concurrent transactions log to debug level - [`10ee8dd`](https://togithub.com/twmb/franz-go/commit/10ee8dd) group consuming: add reasons to JoinGroup, LeaveGroup per KIP-800 - [`0bfaf64`](https://togithub.com/twmb/franz-go/commit/0bfaf64) consumer group: track topics that the leader is not interested in - [`e8495bb`](https://togithub.com/twmb/franz-go/commit/e8495bb) client: add ForceMetadataRefresh - [`c763c9b`](https://togithub.com/twmb/franz-go/commit/c763c9b) consuming: add NoResetOffset - [`4e0e1d7`](https://togithub.com/twmb/franz-go/commit/4e0e1d7) config: add UnknownTopicRetries option, use more widely - [`7f58a97`](https://togithub.com/twmb/franz-go/commit/7f58a97) config: lower default MetadataMinAge to 2.5s - [`e7bd28f`](https://togithub.com/twmb/franz-go/commit/e7bd28f) Client,GroupTransactSession: add TryProduce - [`2a2cf66`](https://togithub.com/twmb/franz-go/commit/2a2cf66) consumer: add AddConsumeTopics - [`d178e26`](https://togithub.com/twmb/franz-go/commit/d178e26) client: add PurgeTopicsFromClient - [`336d2c9`](https://togithub.com/twmb/franz-go/commit/336d2c9) kgo: add ConcurrentTransactionsBackoff, MaxProduceRequestsInflightPerBroker - [`fb04711`](https://togithub.com/twmb/franz-go/commit/fb04711) kversion: cut v3.1 ### [`v1.3.5`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v135) [Compare Source](https://togithub.com/twmb/franz-go/compare/v1.3.4...v1.3.5) \=== This patch release fixes a panic in `GroupTransactSession.End` and has three behavior changes that are beneficial to users. The panic was introduced in v1.3.0; if using a `GroupTransactSession`, it is recommended you upgrade to this release. The next release aims to be v1.4.0, this release is a small one to address a few issues before the much larger and feature filled v1.4 release. - [`010e8e1`](https://togithub.com/twmb/franz-go/commit/010e8e1) txn: fix panic in GroupTransactSession.End - [`f9cd625`](https://togithub.com/twmb/franz-go/commit/f9cd625) consuming: handle exact offset consuming better - [`2ab1978`](https://togithub.com/twmb/franz-go/commit/2ab1978) EndTransaction: return nil rather than an error if not in a transaction - [`96bfe52`](https://togithub.com/twmb/franz-go/commit/96bfe52) broker: remove 5s minimum for sasl session lifetime ### [`v1.3.4`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v134) [Compare Source](https://togithub.com/twmb/franz-go/compare/v1.3.3...v1.3.4) \=== This small patch release fixes a problem with [`4f2e7fe3`](https://togithub.com/twmb/franz-go/commit/4f2e7fe3) which was meant to address [#98](https://togithub.com/twmb/franz-go/issues/98). The fix was not complete in that the fix would only trigger if a group member had partitions added to it. We now rearrange the logic such that it occurs always. This bug was found while making a change in the code in support of a new feature in the v1.4 branch; this bug was not encountered in production. This also bumps the franz-go/pkg/kmsg dependency so that `JoinGroup.Reason` is properly tagged as v9+. The next release will be v1.4, which is nearly ready to be merged into this branch and tagged. Follow issue [#135](https://togithub.com/twmb/franz-go/issues/135) for more details. - [`02560c7`](https://togithub.com/twmb/franz-go/commit/02560c7) consumer group: bugfix fetch offsets spanning rebalance ### [`v1.3.3`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v133) [Compare Source](https://togithub.com/twmb/franz-go/compare/v1.3.2...v1.3.3) \=== This patch release contains two minor bug fixes and a few small behavior changes. The upcoming v1.4 release will contain more changes to lint the entire codebase and will have a few new options as features to configure the client internals. There are a few features in development yet that I would like to complete before tagging the next minor release. #### Bug fixes Repeated integration testing resulted in a rare data race, and one other bug was found by linting. For the race, if a group was left *or* heartbeating stopped *before* offset fetching finished, then there would be a concurrent double write to an error variable: one write would try to write an error from the request being cut (which would be `context.Canceled`), and the other write would write the same error, but directly from `ctx.Err`. Since both of these are the same type pointer and data pointer, it is unlikely this race would result in anything if it was ever encountered, and encountering it would be rare. For the second bug, after this prior one, I wondered if any linter would have caught this bug (the answer is no). However, in the process of heavily linting the code base, a separate bug was found. This bug has **no impact**, but it is good to fix. If you previously consumed and specified *exact* offsets to consume from, the internal `ListOffsets` request would use that offset as a timestamp, rather than using -1 as I meant to internally. The whole point is just to load the partition, so using a random number for a timestmap is just as good as using -1, but we may as well use -1 to be proper. #### Behavior changes Previously when producing, the buffer draining goroutine would sleep for 50ms whenever it started. This was done to give high throughput producers a chance to produce more records within the first batch, rather than the loop being so fast that one record is in the first batch (and more in the others). This 50ms sleep is a huge penalty to oneshot producers (producing one message at a time, synchronously) and also was a slight penalty to high throughput producers whenever the drain loop quit and needed to be resumed. We now eliminate this sleep. This may result in more smaller batches, but definitely helps oneshot producers. A linger can be used to avoid small batches, if needed. Previously, due to MS Azure improperly replying to produce requests when acks were zero, a discard goroutine was added to drain the produce connection if the client was configured to produce with no acks. Logic has been added to quit this goroutine if nothing is read on that goroutine for 3x the connection timeout overhead, which is well enough time that a response should be received if the broker is ever going to send one. Previously, `MarkCommitRecords` would forbid rewinds and only allow advancing offsets. The code now allows rewinds if you mark an early offset after you have already marked a later offset. This brings the behavior in line with the current `CommitOffsets`. Previously, if the client encountered `CONCURRENT_TRANSACTIONS` during transactions, it would sleep for 100ms and then retry the relevant request. This sleep has been dropped to 20ms, which should help latency when transacting quickly. The v1.4 release will allow this number to be configured with a new option. #### Additions KIP-784 and KIP-814 are now supported (unreleased yet in Kafka). Support for KIP-814 required bumping the franz-go's kmsg dep. Internally, only KIP-814 affects client behavior, but since this is unrelased, it is not changing any behavior. #### Relevant commits - [`b39ca31`](https://togithub.com/twmb/franz-go/commit/b39ca31) fetchOffsets: fix data race - [`4156e9f`](https://togithub.com/twmb/franz-go/commit/4156e9f) kgo: fix one bug found by linting - [`72760bf..ad991d8`](https://togithub.com/twmb/franz-go/compare/72760bf..ad991d8) kmsg, kversion, kgo: support KIP-814 (SkipAssignment in JoinGroupResponse) - [`db9017a`](https://togithub.com/twmb/franz-go/commit/db9017a) broker: allow the acks==0 producing discard goroutine to die - [`eefb1f3`](https://togithub.com/twmb/franz-go/commit/eefb1f3) consuming: update docs & simplify; `MarkCommitRecords`: allow rewinds - [`8808b94`](https://togithub.com/twmb/franz-go/commit/8808b94) sink: remove 50ms wait on new drain loops - [`a13f918`](https://togithub.com/twmb/franz-go/commit/a13f918) kmsg & kversion: add support for KIP-784 (ErrorCode in DescribeLogDirs response) - [PR #133](https://togithub.com/twmb/franz-go/pull/133) - lower concurrent transactions retry to 20ms; configurability will be in the next release (thanks [@eduard-netsajev](https://togithub.com/eduard-netsajev)) ### [`v1.3.2`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v132) [Compare Source](https://togithub.com/twmb/franz-go/compare/v1.3.1...v1.3.2) \=== This patch fixes a bug of unclear severity related to transactions. Credit goes to [@eduard-netsajev](https://togithub.com/eduard-netsajev) for finding this long standing problem. In Kafka, if you try to start a transaction too soon after finishing the previous one, Kafka may not actually have internally finished the prior transaction yet and can return a `CONCURRENT_TRANSACTIONS` error. To work around this, clients are expected to retry when they see this error (even though it is marked as not retriable). This client does that properly, but unfortunately did not bubble up any *non* `CONCURRENT_TRANSACTIONS` errors. From the code, it *appears* as if in the worst case, this could have meant that transactions invisibly looked like they were working and being used when they actually were not. However, it's likely that other errors would be noticed internally, and it's possible that if you encountered problems, the entire ETL pipeline would stall anyway. All told, it's not entirely clear what the ramifications for this bug are, and it is recommended that if you use transactions, you should update immediately. - [PR #131](https://togithub.com/twmb/franz-go/pull/131) - txns: don't ignore error in doWithConcurrentTransactions ### [`v1.3.1`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v131) [Compare Source](https://togithub.com/twmb/franz-go/compare/v1.3.0...v1.3.1) \=== This small patch release fixes a leaked-goroutine problem after closing a consuming client, and adds one config validation to hopefully reduce confusion. For the bug, if a fetch was buffered when you were closing the client, an internal goroutine would stay alive. In normal cases where you have one client in your program and you close it on program shutdown, this leak is not really important. If your program recreates consuming clients often and stays alive, the leaked goroutine could eventually result in unexpected memory consumption. Now, the client internally unbuffers all fetches at the end of `Close`, which allows the previously-leaking goroutine to exit. For the config validation, using `ConsumePartitions` and `ConsumeTopics` with the same topic in both options would silently ignore the topic in `ConsumePartitions`. Now, the client will return an error that using the same topic in both options is invalid. - [`ea11266`](https://togithub.com/twmb/franz-go/commit/ea11266) config: add duplicately specified topic validation - [`bb581f4`](https://togithub.com/twmb/franz-go/commit/bb581f4) **bugfix** client: unbuffer fetches on Close to allow mangeFetchConcurrency to quit ### [`v1.3.0`](https://togithub.com/twmb/franz-go/blob/HEAD/CHANGELOG.md#v130) [Compare Source](https://togithub.com/twmb/franz-go/compare/v1.2.6...v1.3.0) \=== This release contains three new features, a few behavior changes, and one minor bugfix. For features, you can now adjust fetched offsets before they are used (thanks [@michaelwilner](https://togithub.com/michaelwilner)!), you can now "ping" your cluster to see if the client can connect at all, and you can now use `SetOffsets` when consuming partitions manually. As a somewhat of a feature-ish, producing no longer requires a context, instead if a context is nil, `context.Background` is used (this was added to allow more laziness when writing small unimportant files). The transactional behavior change is important: the documentation changes are worth reading, and it is worth using a 2.5+ cluster along with the `RequireStableFetchOffsets` option if possible. The metadata leader epoch rewinding behavior change allows the client to continue in the event of odd cluster issues. In kadm, we now return individual per-partition errors if partitions are not included in OffsetCommit responses. The generated code now has a few more enums (thanks [@weeco](https://togithub.com/weeco)!) Lastly, as a small bugfix, `client.Close()` did not properly stop seed brokers. A previous commit split seed brokers and non-seed brokers internally into two fields but did not add broker shutdown on the now-split seed broker field. - [`e0b520c`](https://togithub.com/twmb/franz-go/commit/e0b520c) **behavior change** kadm: set per-partition errors on missing offsets in CommitOffsets - [`32425df`](https://togithub.com/twmb/franz-go/commit/32425df) **feature** client: add Ping method - [`a059901`](https://togithub.com/twmb/franz-go/commit/a059901) **behavior change** txns: sleep 200ms on commit, preventing rebalance / new commit - [`12eaa1e`](https://togithub.com/twmb/franz-go/commit/12eaa1e) **behavior change** metadata: allow leader epoch rewinds after 5 tries - [`029e655`](https://togithub.com/twmb/franz-go/commit/029e655) **feature-ish** Produce{,Sync}: default to context.Background if no ctx is provided - [`eb2cec3`](https://togithub.com/twmb/franz-go/commit/eb2cec3) **bugfix** client: stop seed brokers on client.Close - [`2eae20d`](https://togithub.com/twmb/franz-go/commit/2eae20d) **feature** consumer: allow SetOffsets for direct partition consuming - [pr #120](https://togithub.com/twmb/franz-go/pull/120) **feature** Add groupopt to swizzle offset assignments before consumption (thanks [@michaelwilner](https://togithub.com/michaelwilner)!)Configuration
📅 Schedule: At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR has been generated by WhiteSource Renovate. View repository job log here.