search

uhthomas / automata

Monorepo for Starjunk and subsidiaries
32 stars 2 forks source link

chore(deps): update dependency emqx/emqx to v5.8.1 #494

Open renovate[bot] opened 6 months ago

renovate[bot] commented 6 months ago

This PR contains the following updates:

Package Update Change
emqx/emqx minor 5.6.0 -> 5.8.1

[!WARNING] Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

Release Notes

emqx/emqx (emqx/emqx) ### [`v5.8.1`](https://redirect.github.com/emqx/emqx/releases/tag/v5.8.1): EMQX v5.8.1 [Compare Source](https://redirect.github.com/emqx/emqx/compare/v5.8.0...v5.8.1) ### 5.8.1 *Release Date: 2024-10-14* Make sure to check the breaking changes and known issues before upgrading to EMQX 5.8.1. #### Important Changes - [#​13956](https://redirect.github.com/emqx/emqx/pull/13956) Updated the `gen_rpc` library to version 3.4.1, which includes a node crash issue. Previously, if a node is force shutdown down while RPC channels are being established, it may cause a cluster peer node to crash. #### Enhancements ##### Core MQTT Functionalities - [#​13525](https://redirect.github.com/emqx/emqx/pull/13525) Added new configuration item `shared_subscription_initial_sticky_pick` to specify the strategy for making the initial pick when `shared_subscription_strategy` is set to `sticky`. - [#​13942](https://redirect.github.com/emqx/emqx/pull/13942) The HTTP client now automatically reconnects if no activity is detected for 10 seconds after the latest request has expired. Previously, it would wait indefinitely for a server response, causing timeouts if the server dropped requests. This change impacts below components. - HTTP authentication - HTTP authorization - Webhook (HTTP connector) ##### Authentication and Authorization - [#​13863](https://redirect.github.com/emqx/emqx/pull/13863) EMQX now supports `${cert_common_name}` placeholder in topic name templates for raw ACL rules. - [#​13792](https://redirect.github.com/emqx/emqx/pull/13792) The banned-clients API `GET /banned` supports querying the rules using filters in the query string. The available filters are: - clientid - username - peerhost - like_clientid - like_username - like_peerhost - like_peerhost_net When adding a new banned client entry, the default expiration time for entries without the `until` parameter specified has been changed from 1 year to `infinite`. ##### Rule Engine - [#​13773](https://redirect.github.com/emqx/emqx/pull/13773) Disabled rule actions now do not trigger `out_of_service` warnings. Previously, if an action is disabled, there would be a warning log with `msg: out_of_service`, and the `actions.failed` counter was incremented for the rule. After this enhancement, disabled action will result in a `debug` level log with `msg: discarded`, and the newly introduced counter `actions.discarded` will be incremented. ##### MQTT over QUIC - [#​13814](https://redirect.github.com/emqx/emqx/pull/13814) Connection Scope Keepalive for MQTT over QUIC Multi-Stream: This update introduces a new feature to maintain MQTT connections over QUIC multi-streams, even when the control stream is idle but other data streams are active. Previously, clients had to send `MQTT.PINGREQ` on idle control streams to keep the connection alive. Now, a shared state is maintained for each connection, monitoring activity across all streams. This shared state helps determine if the connection is still active, reducing the risk of keepalive timeouts caused by Head-of-Line (HOL) blocking and improving overall connection stability. #### Bug Fixes ##### Core MQTT Functions - [#​13702](https://redirect.github.com/emqx/emqx/pull/13702) Clean up the corresponding exclusive subscriptions when a node goes down. - [#​13708](https://redirect.github.com/emqx/emqx/pull/13708) Fixed an issue which may cause shared subscription 'sticky' strategy to degrade to 'random'. - [#​13733](https://redirect.github.com/emqx/emqx/pull/13733) Made `cacertfile` optional when configuring https listener from `emqx ctl conf load` command. - [#​13742](https://redirect.github.com/emqx/emqx/pull/13742) Fixed when subscribing with `+` as the first level, or `#` as a wildcard, retained messages with topics starting with `$` are incorrectly received. - [#​13754](https://redirect.github.com/emqx/emqx/pull/13754) Fixed an issue when websocket connection would break consistently on its own. - [#​13756](https://redirect.github.com/emqx/emqx/pull/13756) Introduced more randomness to broker assigned client IDs. - [#​13790](https://redirect.github.com/emqx/emqx/pull/13790) The default heartbeat interval for the MQTT connector has been reduced from 300 seconds to 160 seconds. This change helps maintain the underlying TCP connection by preventing timeouts due to the idle limits imposed by load balancers or firewalls, which typically range from 3 to 5 minutes depending on the cloud provider. - [#​13832](https://redirect.github.com/emqx/emqx/pull/13832) Fixed that the `Publish` endpoint would have a 500 error when persistent session were enabled. - [#​13842](https://redirect.github.com/emqx/emqx/pull/13842) Fixed a UTF-8 string validation exception. ##### Upgrade and Migration - [#​13731](https://redirect.github.com/emqx/emqx/pull/13731) Resolved an issue that prevented clusters running on EMQX 5.4.0 from upgrading to EMQX 5.8.0. This fix introduces a migration procedure to update specific internal database tables created in version 5.4.0 to align with the new schema. #### Breaking Changes - [#​13792](https://redirect.github.com/emqx/emqx/pull/13792) The default expiration time for a banned item that is created without an `until` value is now `infinity` (previsouly capped at 1 year limit). - [#​13742](https://redirect.github.com/emqx/emqx/pull/13742) Fixed an issue when a client would receive retained messages for a topic starting with `$` when it subscribed to topic `#` or `+`. This fix satisfies the requirement of [MQTT-4.7.2-1](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#\_Toc3901246). ### [`v5.8.0`](https://redirect.github.com/emqx/emqx/releases/tag/v5.8.0): EMQX v5.8.0 [Compare Source](https://redirect.github.com/emqx/emqx/compare/v5.7.2...v5.8.0) #### Enhancements ##### Core MQTT Functionality - [#​13009](https://redirect.github.com/emqx/emqx/pull/13009) Updated the log level for message receiving pause due to rate limiting from `debug` to `warning`. The log message `socket_receive_paused_by_rate_limit` is throttled to avoid excessive logging. ##### Authentication and Authorization - [#​12418](https://redirect.github.com/emqx/emqx/pull/12418) Enhanced JWT authentication to support claims verification using a list of objects: [ { name = "claim_name", value = "${username}" }, ... ] Expected values are now treated as templates, consistent with other authenticators, allowing for arbitrary expressions such as `${username}` and `${clientid}`. Previousy, only fixed `"${username}"` `"${clientid}"` values were supported for interpolation. Improved the documentation for the `verify_claims` parameter. - [#​13229](https://redirect.github.com/emqx/emqx/pull/13229) Added support for `${cert_pem}` placeholder in authentication templates. - [#​13534](https://redirect.github.com/emqx/emqx/pull/13534) Added trace logging to indicate when the superuser bypasses the authorization check. ##### Data Integrations - [#​13144](https://redirect.github.com/emqx/emqx/pull/13144) Changed the log level to `warning` and added throttling for the log message `data_bridge_buffer_overflow` when bridge buffers overflow and messages are dropped. Previously, these events were logged at the `info` level and were not visible with the default log settings. - [#​13492](https://redirect.github.com/emqx/emqx/pull/13492) Enhanced the `GET /connectors` and `GET /connectors/:id` APIs to include lists of actions and sources that depend on a specific connector. Additionally, the `GET /actions`, `GET /sources`, `GET /actions/:id`, and `GET /sources/:id` APIs now return the list of rules associated with a specific action or source. - [#​13505](https://redirect.github.com/emqx/emqx/pull/13505) Added the ability to filter rules in the HTTP API based on the IDs of data integration actions or sources used. - [#​13506](https://redirect.github.com/emqx/emqx/pull/13506) Introduced the `peername` field to all rule engine events that already include the `peerhost` field. The `peername` field is a string formatted as `IP:PORT`. - [#​13516](https://redirect.github.com/emqx/emqx/pull/13516) Added a `direct_dispatch` argument to the `republish` action. When `direct_dispatch` is set to `true` (or rendered as `true` from template) the message is dispatched directly to subscribers. This feature helps prevent the triggering of additional rules or the recursive activation of the same rule. - [#​13573](https://redirect.github.com/emqx/emqx/pull/13573) Introduced `client_attrs` to the SQL context for client connectivity events and the message `publish` event. Users can now access client attributes within rule SQL statements, such as `SELECT client_attrs.attr1 AS attribute1`, and utilize `${attribute1}` in data integration actions. - [#​13640](https://redirect.github.com/emqx/emqx/pull/13640) Added two new SQL functions for rules: `coalesce/2` and `coalesce_ne/2`. These functions simplify handling null values in rule SQL expressions. For instance, instead of using: SELECT CASE WHEN is_null(payload.path.to.value) THEN 0 ELSE payload.path.to.value END AS my_value you can now write a more concise expression: `SELECT coalesce(payload.path.to.value, 0) AS my_value`. ##### Operations - [#​13202](https://redirect.github.com/emqx/emqx/pull/13202) Introduced the `emqx ctl conf cluster_sync fix` command to address cluster configuration inconsistencies. This command synchronizes the configuration of all nodes with the configuration of the node that has the highest `tnx_id`, ensuring consistency across the cluster. - [#​13250](https://redirect.github.com/emqx/emqx/pull/13250) Added a new value for `cluster.discovery_strategy`: `singleton`. By choosing this option, there will be effectively no clustering, and the node will reject connection attempts to and from other nodes. - [#​13370](https://redirect.github.com/emqx/emqx/pull/13370) Added a new version of `wildcard_optimized` storage layout for durable storage, offering the following improvements: - The new layout does not have an inherent latency. - MQTT messages are serialized into a more space-efficient format. - [#​13524](https://redirect.github.com/emqx/emqx/pull/13524) Added the `emqx ctl exclusive` CLI interface to manage exclusive topics more effectively. It allows administrators to better manage and troubleshoot exclusive topic subscriptions, ensuring that subscription states are accurately reflected and preventing unexpected failures. - [#​13597](https://redirect.github.com/emqx/emqx/pull/13597) Added thin wrapper functions for plugins to store and manage the certificate files used by the plugins themselves. This fix prevents plugin certificates from being inadvertently deleted by the certificate garbage collection (GC) function. - [#​13626](https://redirect.github.com/emqx/emqx/pull/13626) Added a new command `emqx ctl listeners enable ` to enable/disable a listener. - [#​13493](https://redirect.github.com/emqx/emqx/pull/13493) Upgraded the RPC library `gen_rpc` to version 3.4.0. This update changes the default RPC server socket option from `true` to `active-100`, which introduces back-pressure to peer nodes when the RPC server experiences heavy load. - [#​13665](https://redirect.github.com/emqx/emqx/pull/13665) Added a new metric `emqx_actions_count` to the prometheus endpoint. It contains the number of all actions added by all rules, including Republish actions and Console Output actions. #### Bug Fixes #### Core MQTT Functionality - [#​12944](https://redirect.github.com/emqx/emqx/pull/12944) Fixed an issue that caused a crash when clients with non-UTF8 client IDs attempted to connect with `strict_mode=false`. - [#​13006](https://redirect.github.com/emqx/emqx/pull/13006) Improved the validation of retained, delayed, and taken-over session messages to ensure they comply with banned client ID rules implemented through regular expression matching. Previously, certain messages, such as those delayed due to network issues or taken over by another session, could bypass the client ID bans set by regular expressions. ##### Authentication and Authorization - [#​13024](https://redirect.github.com/emqx/emqx/pull/13024) Added a default ACL deny rule to reject subscriptions to the `+/#` topic pattern. Since EMQX by default rejects subscriptions to `#` topic, for completeness, it should reject `+/#` as well. - [#​13040](https://redirect.github.com/emqx/emqx/pull/13040) Improved HTTP authentication: - Improved error logging for cases where the HTTP `Content-Type` header is missing or unrecognized, providing more detailed information. - Fixed an issue causing double encoding of query parameters in authentication HTTP requests - Enhanced error messages when a POST method with a JSON content type is configured for authentication requests but the JSON template fails to render into valid JSON. This can occur, for example, when a template contains a placeholder like `${password}` but receives a non-UTF8 password input, leading to better transparency and easier debugging for such scenarios. - [#​13196](https://redirect.github.com/emqx/emqx/pull/13196) Added a limit to the built-in authorization database, restricting the number of Access Control List (ACL) rules per client or user to a default of 100. - [#​13584](https://redirect.github.com/emqx/emqx/pull/13584) Fixed an issue with creating HTTP authorization that resulted in errors when the HTTP header list was empty. - [#​13618](https://redirect.github.com/emqx/emqx/pull/13618) Improved the type specifications for the `authorization/sources` endpoint to provide clearer and more concise error messages. - [#​13624](https://redirect.github.com/emqx/emqx/pull/13624) Fixed an issue in the built-in authorizer where updating rules for a client or user could result in the total number of rules exceeding the `max_rules` limit. - [#​13678](https://redirect.github.com/emqx/emqx/pull/13678) Made the deletion of an authenticator in the chain an idempotent operation, ensuring that deleting a non-existing authenticator always succeeds. ##### Data Integrations - [#​13207](https://redirect.github.com/emqx/emqx/pull/13207) Improved the `republish` rule engine action to accurately reflect the success and failure of message publishing. Previously, the success metrics were incremented even when the republish action failed to deliver the message to any subscribers. Now, if the action detects that a message fails to reach any subscriber, the failure metrics are correctly incremented. - [#​13425](https://redirect.github.com/emqx/emqx/pull/13425) Improved the MQTT connector error log messages to provide clearer and more detailed information. - [#​13589](https://redirect.github.com/emqx/emqx/pull/13589) Fixed an issue where creating a rule with a string `"null"` for ID via the HTTP API was allowed, which could lead to an inconsistent configuration. ##### Operations - [#​13078](https://redirect.github.com/emqx/emqx/pull/13078) Improved validation and error handling in the EMQX Management API to ensure that requests with a JSON body include the `Content-Type: application/json` header. If the header is missing for APIs that expect JSON input, the server now correctly responds with a `415 Unsupported Media Type` status code instead of `400 Bad Request`. - [#​13225](https://redirect.github.com/emqx/emqx/pull/13225) Enhanced security in authentication and authorization APIs by redacting sensitive data such as passwords. Previously, the APIs could return the original password values in responses. With this update, sensitive information is replaced with `******` to prevent accidental exposure and protect user credentials. ##### Gateways - [#​13607](https://redirect.github.com/emqx/emqx/pull/13607) Fixed an issue where the QoS level for CoAP subscriptions displayed through the API did not match the actual QoS level being used. This discrepancy could cause confusion as successful subscriptions were not accurately reflected on the Dashboard. #### Breaking changes - [#​13526](https://redirect.github.com/emqx/emqx/pull/13526) Removed the Core-replicant feature from the Open-Source Edition. Starting from release 5.8, all nodes running the Open-Source Edition will operate in the Core role. This change does not impact Enterprise Edition users, who will continue to have access to the Core-replicant functionality. Additionally, the obsolete `cluster.core_nodes` configuration parameter has been removed as it is no longer needed. - **Dashboard Updates**: The following features have been removed or restricted in the Open-Source Edition Dashboard: - Monitoring: - Delayed Publish - Alarms - Access Control: - Authentication (LDAP) - Authorization (LDAP) - Flapping Detect - Integration: - Flow Designer - Management: - Monitoring - Advanced MQTT - Topic Rewrite - Auto Subscribe - Delayed Publish - Diagnose: - Topic Metrics - Slow Subscriptions - [#​13080](https://redirect.github.com/emqx/emqx/pull/13080) Updated the default value of the `mqtt.retry_interval` configuration from 30 seconds to `infinity`. Previously, EMQX would automatically retry message deliveries every 30 seconds by default. With the new default set to `infinity`, EMQX will no longer retry message deliveries automatically. This change aligns with MQTT specification standards, which generally do not recommend in-session message delivery retries. We understand that some users rely on the retry feature, so the ability to configure a specific retry interval is still available for backward compatibility. - [#​13190](https://redirect.github.com/emqx/emqx/pull/13190) Discontinued support for releases on CentOS 7 and Ubuntu 18. EMQX will no longer provide builds for these operating systems due to their end-of-life status. - [#​13248](https://redirect.github.com/emqx/emqx/pull/13248) Replaced the `builtin` durable storage backend with two new backends to provide better flexibility and scalability: - **`builtin_local`**: A durable storage backend that does not support replication, making it suitable for single-node deployments. This backend is available in both the open-source and enterprise editions of EMQX but is not compatible with multi-node clusters. - **`builtin_raft`**: A durable storage backend utilizing the Raft consensus algorithm for data replication across multiple nodes. This backend is exclusively available in the enterprise edition of EMQX, providing enhanced data durability and fault tolerance. Additionally, several Prometheus metrics have been renamed to better reflect their functions: - `emqx_ds_egress_batches` has been renamed to `emqx_ds_buffer_batches` - `emqx_ds_egress_batches_retry` has been renamed to `emqx_ds_buffer_batches_retry` - `emqx_ds_egress_batches_failed` has been renamed to `emqx_ds_buffer_batches_failed` - `emqx_ds_egress_messages` has been renamed to `emqx_ds_buffer_messages` - `emqx_ds_egress_bytes` has been renamed to `emqx_ds_buffer_bytes` - `emqx_ds_egress_flush_time` has been renamed to `emqx_ds_buffer_flush_time` ### [`v5.7.2`](https://redirect.github.com/emqx/emqx/releases/tag/v5.7.2): EMQX v5.7.2 [Compare Source](https://redirect.github.com/emqx/emqx/compare/v5.7.1...v5.7.2) #### Enhancements - [#​13317](https://redirect.github.com/emqx/emqx/pull/13317) Added a new per-authorization source metric type: `ignore`. This metric increments when an authorization source attempts to authorize a request but encounters scenarios where the authorizer is not applicable or encounters an error, resulting in an undecidable outcome. - [#​13336](https://redirect.github.com/emqx/emqx/pull/13336) Added functionality to initialize authentication data in the built-in database of an empty EMQX node or cluster using a bootstrap file in CSV or JSON format. This feature introduces new configuration entries, `bootstrap_file` and `bootstrap_type`. - [#​13348](https://redirect.github.com/emqx/emqx/pull/13348) Added a new field `payload_encode` in the log configuration to determine the format of the payload in the log data. - [#​13436](https://redirect.github.com/emqx/emqx/pull/13436) Added the option to add custom request headers to JWKS requests. - [#​13507](https://redirect.github.com/emqx/emqx/pull/13507) Introduced a new built-in function `getenv` in the rule engine and variform expression to facilitate access to environment variables. This function adheres to the following constraints: - Prefix `EMQXVAR_` is added before reading from OS environment variables. For example, `getenv('FOO_BAR')` is to read `EMQXVAR_FOO_BAR`. - These values are immutable once loaded from the OS environment. - [#​13521](https://redirect.github.com/emqx/emqx/pull/13521) Resolved an issue where LDAP query timeouts could cause the underlying connection to become unusable, potentially causing subsequent queries to return outdated results. The fix ensures the system reconnects automatically in case of a timeout. - [#​13528](https://redirect.github.com/emqx/emqx/pull/13528) Applied log throttling for the event of unrecoverable errors in data integrations. - [#​13548](https://redirect.github.com/emqx/emqx/pull/13548) EMQX now can optionally invoke the `on_config_changed/2` callback function when the plugin configuration is updated via the REST API. This callback function is assumed to be exported by the `_app` module. For example, if the plugin name and version are `my_plugin-1.0.0`, then the callback function is assumed to be `my_plugin_app:on_config_changed/2`. - [#​13386](https://redirect.github.com/emqx/emqx/pull/13386) Added support for initializing a list of banned clients on an empty EMQX node or cluster with a bootstrap file in CSV format. The corresponding config entry to specify the file path is `banned.bootstrap_file`. This file is a CSV file with `,` as its delimiter. The first line of this file must be a header line. All valid headers are listed here: - as :: required - who :: required - by :: optional - reason :: optional - at :: optional - until :: optional See the [Configuration Manual](https://docs.emqx.com/en/enterprise/v@EE_VERSION@/hocon/) for details on each field. Each row in the rest of this file must contain the same number of columns as the header line, and the column can be omitted then its value is `undefined`. #### Bug Fixes - [#​13222](https://redirect.github.com/emqx/emqx/pull/13222) Resolved issues with flags checking and error handling associated with the Will message in the `CONNECT` packet. For detailed specifications, refer to: - MQTT-v3.1.1-\[MQTT-3.1.2-13], MQTT-v5.0-\[MQTT-3.1.2-11] - MQTT-v3.1.1-\[MQTT-3.1.2-14], MQTT-v5.0-\[MQTT-3.1.2-12] - MQTT-v3.1.1-\[MQTT-3.1.2-15], MQTT-v5.0-\[MQTT-3.1.2-13] - [#​13307](https://redirect.github.com/emqx/emqx/pull/13307) Updated `ekka` library to version 0.19.5. This version of `ekka` utilizes `mria` 0.8.8, enhancing auto-heal functionality. Previously, the auto-heal worked only when all core nodes were reachable. This update allows to apply auto-heal once the majority of core nodes are alive. For details, refer to the [Mria PR](https://redirect.github.com/emqx/mria/pull/180). - [#​13334](https://redirect.github.com/emqx/emqx/pull/13334) Implemented strict mode checking for the `PasswordFlag` in the MQTT v3.1.1 CONNECT packet to align with protocol specifications. Note: To ensure bug-to-bug compatibility, this check is performed only in strict mode. - [#​13344](https://redirect.github.com/emqx/emqx/pull/13344) Resolved an issue where the `POST /clients/:clientid/subscribe/bulk` API would not function correctly if the node receiving the API request did not maintain the connection to the specified `clientid`. - [#​13358](https://redirect.github.com/emqx/emqx/pull/13358) Fixed an issue when the `reason` in the `authn_complete_event` event was incorrectly displayed. - [#​13375](https://redirect.github.com/emqx/emqx/pull/13375) The value `infinity` has been added as default value to the listener configuration fields `max_conn_rate`, `messages_rate`, and `bytes_rate`. - [#​13382](https://redirect.github.com/emqx/emqx/pull/13382) Updated the `emqtt` library to version 0.4.14, which resolves an issue preventing `emqtt_pool`s from reusing pools that are in an inconsistent state. - [#​13389](https://redirect.github.com/emqx/emqx/pull/13389) Fixed an issue where the `Derived Key Length` for `pbkdf2` could be set to a negative integer. - [#​13389](https://redirect.github.com/emqx/emqx/pull/13389) Fixed an issue where topics in the authorization rules might be parsed incorrectly. - [#​13393](https://redirect.github.com/emqx/emqx/pull/13393) Fixed an issue where plugin applications failed to restart after a node joined a cluster, resulting in hooks not being properly installed and causing inconsistent states. - [#​13398](https://redirect.github.com/emqx/emqx/pull/13398) Fixed an issue where ACL rules were incorrectly cleared when reloading the built-in database for authorization using the command line. - [#​13403](https://redirect.github.com/emqx/emqx/pull/13403) Addressed a security issue where environment variable configuration overrides were inadvertently logging passwords. This fix ensures that passwords present in environment variables are not logged. - [#​13408](https://redirect.github.com/emqx/emqx/pull/13408) Resolved a `function_clause` crash triggered by authentication attempts with invalid salt or password types. This fix enhances error handling to better manage authentication failures involving incorrect salt or password types. - [#​13419](https://redirect.github.com/emqx/emqx/pull/13419) Resolved an issue where crash log messages from the `/configs` API were displaying garbled hints. This fix ensures that log messages related to API calls are clear and understandable. - [#​13422](https://redirect.github.com/emqx/emqx/pull/13422) Fixed an issue where the option `force_shutdown.max_heap_size` could not be set to 0 to disable this tuning. - [#​13442](https://redirect.github.com/emqx/emqx/pull/13442) Fixed an issue where the health check interval configuration for actions/sources was not being respected. Previously, EMQX ignored the specified health check interval for actions and used the connector's interval instead. The fix ensures that EMQX now correctly uses the health check interval configured for actions/sources, allowing for independent and accurate health monitoring frequencies. - [#​13503](https://redirect.github.com/emqx/emqx/pull/13503) Fixed an issue where connectors did not adhere to the configured health check interval upon initial startup, requiring an update or restart to apply the correct interval. - [#​13515](https://redirect.github.com/emqx/emqx/pull/13515) Fixed an issue where the same client could not subscribe to the same exclusive topic when the node was down for some reason. - [#​13527](https://redirect.github.com/emqx/emqx/pull/13527) Fixed an issue in the Rule Engine where executing a SQL test for the Message Publish event would consistently return no results when a `$bridges/...` source was included in the `FROM` clause. - [#​13541](https://redirect.github.com/emqx/emqx/pull/13541) Fixed an issue where disabling CRL checks for a listener required a listener restart to take effect. - [#​13552](https://redirect.github.com/emqx/emqx/pull/13552) Added a startup timeout limit for EMQX plugins with a default timeout of 10 seconds. Before this update, problematic plugins could cause runtime errors during startup, leading to potential issues where the main startup process might hang when EMQX is stopped and restarted. #### Breaking Changes - [#​12947](https://redirect.github.com/emqx/emqx/pull/12947) For JWT authentication, a new boolean option `disconnect_after_expire` has been added with default value set to `true`. When enabled, the client will be disconnected after the JWT token expires. Previously, the clients with actual JWTs could connect to the broker and stay connected even after the JWT token expired. Now, the client will be disconnected after the JWT token expires. To preserve the previous behavior, set `disconnect_after_expire` to `false`. ### [`v5.7.1`](https://redirect.github.com/emqx/emqx/releases/tag/v5.7.1): EMQX v5.7.1 [Compare Source](https://redirect.github.com/emqx/emqx/compare/v5.7.0...v5.7.1) #### Enhancements - [#​12983](https://redirect.github.com/emqx/emqx/pull/12983) Add new rule engine event `$events/client_check_authn_complete` for authentication completion event. - [#​13180](https://redirect.github.com/emqx/emqx/pull/13180) Improved client message handling performance when EMQX is running on Erlang/OTP 26 and increased message throughput by 10% in fan-in mode. - [#​13191](https://redirect.github.com/emqx/emqx/pull/13191) Upgraded EMQX Docker images to run on Erlang/OTP 26. EMQX had been running on Erlang/OTP 26 since v5.5 except for docker images which were on Erlang/OTP 25. Now all releases are on Erlang/OTP 26. - [#​13242](https://redirect.github.com/emqx/emqx/pull/13242) Significantly increased the startup speed of EMQX dashboard listener. #### Bug Fixes - [#​13156](https://redirect.github.com/emqx/emqx/pull/13156) Resolved an issue where the Dashboard Monitoring pages would crash following the update to EMQX v5.7.0. - [#​13164](https://redirect.github.com/emqx/emqx/pull/13164) Fixed HTTP authorization request body encoding. Before this fix, the HTTP authorization request body encoding format was taken from the `accept` header. The fix is to respect the `content-type` header instead. Also added `access` templating variable for v4 compatibility. The access code of SUBSCRIBE action is `1` and PUBLISH action is `2`. - [#​13238](https://redirect.github.com/emqx/emqx/pull/13238) Improved the logged error messages when an HTTP authorization request with an unsupported content-type header is returned. - [#​13258](https://redirect.github.com/emqx/emqx/pull/13258) Fix an issue where the MQTT-SN gateway would not restart correctly due to incorrect startup order of gateway dependencies. - [#​13273](https://redirect.github.com/emqx/emqx/pull/13273) Fixed and improved handling of URIs in several configurations. The fix includes the following improvement details: - Authentication and authorization configurations: Corrected a previous error where valid pathless URIs such as `https://example.com?q=x` were mistakenly rejected. These URIs are now properly recognized as valid. - Connector configurations: Enhanced checks to ensure that URIs with potentially problematic components, such as user info or fragment parts, are no longer erroneously accepted. - [#​13276](https://redirect.github.com/emqx/emqx/pull/13276) Fixed an issue in the durable message storage mechanism where parts of the internal storage state were not correctly persisted during the setup of new storage generations. The concept of "generation" is used internally and is crucial for managing message expiration and cleanup. This could have manifested as messages being lost after a restart of EMQX. - [#​13291](https://redirect.github.com/emqx/emqx/pull/13291) Fixed an issue where durable storage sites that were down being reported as up. - [#​13290](https://redirect.github.com/emqx/emqx/pull/13290) Fixed an issue where the command `$ bin/emqx ctl rules show rule_0hyd` would produce no output when used to display rules with a data integration action attached. - [#​13293](https://redirect.github.com/emqx/emqx/pull/13293) Improved the restoration process from data backups by automating the re-indexing of imported retained messages. Previously, re-indexing required manual intervention using the `emqx ctl retainer reindex start` CLI command after importing a data backup file. This fix also extended the functionality to allow exporting retained messages to a backup file when the `retainer.backend.storage_type` is configured as `ram`. Previously, only setups with `disc` as the storage type supported exporting retained messages. - [#​13140](https://redirect.github.com/emqx/emqx/pull/13140) Fixed an issue that caused text traces for the republish action to crash and not display correctly. - [#​13148](https://redirect.github.com/emqx/emqx/pull/13148) Fixed an issue where a 500 HTTP status code could be returned by `/connectors/:connector-id/start` when there is a timeout waiting for the resource to be connected. - [#​13181](https://redirect.github.com/emqx/emqx/pull/13181) EMQX now forcefully shut down the connector process when attempting to stop a connector, if such operation times out. This fix also improved the clarity of error messages when disabling an action or source fails due to an unresponsive underlying connector. - [#​13216](https://redirect.github.com/emqx/emqx/pull/13216) Respect `clientid_prefix` config for MQTT bridges. Since EMQX v5.4.1, the MQTT client IDs are restricted to a maximum of 23 bytes. Previously, the system factored the `clientid_prefix` into the hash of the original, longer client ID, affecting the final shortened ID. The fix includes the following change details: - Without Prefix: The behavior remains unchanged. EMQX hashes the long client IDs (exceeding 23 bytes) to fit within the 23-byte limit. - With Prefix: - Prefix ≤ 19 bytes: The prefix is retained, and the remaining portion of the client ID is hashed into a 4-byte space, ensuring the total length does not exceed 23 bytes. - Prefix ≥ 20 bytes: EMQX will not attempt to shorten the client ID, fully preserving the configured prefix regardless of length. ### [`v5.7.0`](https://redirect.github.com/emqx/emqx/releases/tag/v5.7.0): EMQX v5.7.0 [Compare Source](https://redirect.github.com/emqx/emqx/compare/e5.6.1-patch.2...v5.7.0) ##### Enhancements ##### Security - [#​12947](https://redirect.github.com/emqx/emqx/pull/12947) For JWT authentication, support new `disconnect_after_expire` option. When enabled, the client will be disconnected after the JWT token expires. Note: This is a breaking change. This option is enabled by default, so the default behavior is changed. Previously, the clients with actual JWTs could connect to the broker and stay connected even after the JWT token expired. Now, the client will be disconnected after the JWT token expires. To preserve the previous behavior, set `disconnect_after_expire` to `false`. ##### Data Processing and Integration - [#​12671](https://redirect.github.com/emqx/emqx/pull/12671) An `unescape` function has been added to the rule engine SQL language to handle the expansion of escape sequences in strings. This addition has been done because string literals in the SQL language don't support any escape codes (e.g., `\n` and `\t`). This enhancement allows for more flexible string manipulation within SQL expressions. ##### Extensibility - [#​12872](https://redirect.github.com/emqx/emqx/pull/12872) Implemented Client Attributes feature. It allows setting additional properties for each client using key-value pairs. Property values can be generated from MQTT client connection information (such as username, client ID, TLS certificate) or set from data accompanying successful authentication returns. Properties can be used in EMQX for authentication, authorization, data integration, and MQTT extension functions. Compared to using static properties like client ID directly, client properties offer greater flexibility in various business scenarios, simplifying the development process and enhancing adaptability and efficiency in development work. **Initialization of `client_attrs`** The `client_attrs` fields can be initially populated from one of the following `clientinfo` fields: - `cn`: The common name from the TLS client's certificate. - `dn`: The distinguished name from the TLS client's certificate, that is, the certificate "Subject". - `clientid`: The MQTT client ID provided by the client. - `username`: The username provided by the client. - `user_property`: Extract a property value from 'User-Property' of the MQTT CONNECT packet. **Extension through Authentication Responses** Additional attributes may be merged into `client_attrs` from authentication responses. Supported authentication backends include: - **HTTP**: Attributes can be included in the JSON object of the HTTP response body through a `client_attrs` field. - **JWT**: Attributes can be included via a `client_attrs` claim within the JWT. **Usage in Authentication and Authorization** If `client_attrs` is initialized before authentication, it can be used in external authentication requests. For instance, `${client_attrs.property1}` can be used within request templates directed at an HTTP server for authenticity validation. - The `client_attrs` can be utilized in authorization configurations or request templates, enhancing flexibility and control. Examples include: In `acl.conf`, use `{allow, all, all, ["${client_attrs.namespace}/#"]}` to apply permissions based on the `namespace` attribute. - In other authorization backends, `${client_attrs.namespace}` can be used within request templates to dynamically include client attributes. - [#​12910](https://redirect.github.com/emqx/emqx/pull/12910) Added plugin configuration management and schema validation. For EMQX enterprise edition, one can also annotate the schema with metadata to facilitate UI rendering in the Dashboard. See more details in the [plugin template](https://redirect.github.com/emqx/emqx-plugin-template/pull/126) and plugin [documentation](../extensions/plugins.md). ##### Operations and Management - [#​12923](https://redirect.github.com/emqx/emqx/pull/12923) Provided more specific error when importing wrong format into builtin authenticate database. - [#​12940](https://redirect.github.com/emqx/emqx/pull/12940) Added `ignore_readonly` argument to `PUT /configs` API. Before this change, EMQX would return 400 (BAD_REQUEST) if the raw config included read-only root keys (`cluster`, `rpc`, and `node`). After this enhancement it can be called as `PUT /configs?ignore_readonly=true`, EMQX will in this case ignore readonly root config keys, and apply the rest. For observability purposes, an info level message is logged if any readonly keys are dropped. Also fixed an exception when config has bad HOCON syntax (returns 500). Now bad syntax will cause the API to return 400 (BAD_REQUEST). - [#​12957](https://redirect.github.com/emqx/emqx/pull/12957) Started building packages for macOS 14 (Apple Silicon) and Ubuntu 24.04 Noble Numbat (LTS). ##### Bug Fixes ##### Security - [#​12887](https://redirect.github.com/emqx/emqx/pull/12887) Fixed MQTT enhanced auth with sasl scram. - [#​12962](https://redirect.github.com/emqx/emqx/pull/12962) TLS clients can now verify server hostname against wildcard certificate. For example, if a certificate is issued for host `*.example.com`, TLS clients is able to verify server hostnames like `srv1.example.com`. ##### MQTT - [#​12996](https://redirect.github.com/emqx/emqx/pull/12996) Fixed process leak in `emqx_retainer` application. Previously, client disconnection while receiving retained messages could cause a process leak. ##### Data Processing and Integration - [#​12653](https://redirect.github.com/emqx/emqx/pull/12653) The rule engine function `bin2hexstr` now supports bitstring inputs with a bit size that is not divisible by 8. Such bitstrings can be returned by the rule engine function `subbits`. - [#​12657](https://redirect.github.com/emqx/emqx/pull/12657) The rule engine SQL-based language previously did not allow putting any expressions as array elements in array literals (only constants and variable references were allowed). This has now been fixed so that one can use any expressions as array elements. The following is now permitted, for example: ```bash select [21 + 21, abs(-abs(-2)), [1 + 1], 4] as my_array from "t/#" ``` - [#​12932](https://redirect.github.com/emqx/emqx/pull/12932) Previously, if a HTTP action request received a 503 (Service Unavailable) status, it was marked as a failure and the request was not retried. This has now been fixed so that the request is retried a configurable number of times. - [#​12948](https://redirect.github.com/emqx/emqx/pull/12948) Fixed an issue where sensitive HTTP header values like `Authorization` would be substituted by `******` after updating a connector. - [#​13118](https://redirect.github.com/emqx/emqx/pull/13118) Fix a performance issue in the rule engine template rendering. ##### Observability - [#​12765](https://redirect.github.com/emqx/emqx/pull/12765) Make sure stats `subscribers.count` `subscribers.max` contains shared-subscribers. It only contains non-shared subscribers previously. ##### Operations and Management - [#​12812](https://redirect.github.com/emqx/emqx/pull/12812) Made resource health checks non-blocking operations. This means that operations such as updating or removing a resource won't be blocked by a lengthy running health check. - [#​12830](https://redirect.github.com/emqx/emqx/pull/12830) Made channel (action/source) health checks non-blocking operations. This means that operations such as updating or removing an action/source data integration won't be blocked by a lengthy running health check. - [#​12993](https://redirect.github.com/emqx/emqx/pull/12993) Fixed listener config update API when handling an unknown zone. Before this fix, when a listener config is updated with an unknown zone, for example `{"zone": "unknown"}`, the change would be accepted, causing all clients to crash whens connected. After this fix, updating the listener with an unknown zone name will get a "Bad request" response. - [#​13012](https://redirect.github.com/emqx/emqx/pull/13012) The MQTT listerners config option `access_rules` has been improved in the following ways: - The listener no longer crash with an incomprehensible error message if a non-valid access rule is configured. Instead a configuration error is generated. - One can now add several rules in a single string by separating them by comma (for example, "allow 10.0.1.0/24, deny all"). - [#​13041](https://redirect.github.com/emqx/emqx/pull/13041) Improved HTTP authentication error log message. If HTTP content-type header is missing for POST method, it now emits a meaningful error message instead of a less readable exception with stack trace. - [#​13077](https://redirect.github.com/emqx/emqx/pull/13077) This fix makes EMQX only read action configurations from the global configuration when the connector starts or restarts, and instead stores the latest configurations for the actions in the connector. Previously, updates to action configurations would sometimes not take effect without disabling and enabling the action. This means that an action could sometimes run with the old (previous) configuration even though it would look like the action configuration has been updated successfully. - [#​13090](https://redirect.github.com/emqx/emqx/pull/13090) Attempting to start an action or source whose connector is disabled will no longer attempt to start the connector itself. ##### Gateways - [#​12909](https://redirect.github.com/emqx/emqx/pull/12909) Fixed UDP listener process handling on errors or closure, The fix ensures the UDP listener is cleanly stopped and restarted as needed if these error conditions occur. - [#​13001](https://redirect.github.com/emqx/emqx/pull/13001) Fixed an issue where the syskeeper forwarder would never reconnect when the connection was lost. - [#​13010](https://redirect.github.com/emqx/emqx/pull/13010) Fixed the issue where the JT/T 808 gateway could not correctly reply to the REGISTER_ACK message when requesting authentication from the registration service failed. ##### Breaking Changes - [#​12947](https://redirect.github.com/emqx/emqx/pull/12947) For JWT authentication, a new boolean option `disconnect_after_expire` has been added with default value set to `true`. When enabled, the client will be disconnected after the JWT token expires. Previously, the clients with actual JWTs could connect to the broker and stay connected even after the JWT token expired. Now, the client will be disconnected after the JWT token expires. To preserve the previous behavior, set `disconnect_after_expire` to `false`. - [#​12957](https://redirect.github.com/emqx/emqx/pull/12957) Stopped building packages for macOS 12. ### [`v5.6.1`](https://redirect.github.com/emqx/emqx/releases/tag/v5.6.1): EMQX v5.6.1 [Compare Source](https://redirect.github.com/emqx/emqx/compare/v5.6.0...e5.6.1-patch.2) ##### Bug Fixes - [#​12759](https://redirect.github.com/emqx/emqx/pull/12759) EMQX now automatically removes invalid backup files that fail during upload due to schema validation errors. This fix ensures that only valid configuration files are displayed and stored, enhancing system reliability. - [#​12766](https://redirect.github.com/emqx/emqx/pull/12766) Renamed `message_queue_too_long` error reason to `mailbox_overflow` `mailbox_overflow`. The latter is consistent with the corresponding config parameter: `force_shutdown.max_mailbox_size`. - [#​12773](https://redirect.github.com/emqx/emqx/pull/12773) Upgraded HTTP client libraries. The HTTP client library (`gun-1.3`) incorrectly appended a `:portnumber` suffix to the `Host` header for standard ports (`http` on port 80, `https` on port 443). This could cause compatibility issues with servers or gateways performing strict `Host` header checks (e.g., AWS Lambda, Alibaba Cloud HTTP gateways), leading to errors such as `InvalidCustomDomain.NotFound` or "The specified CustomDomain does not exist." - [#​12802](https://redirect.github.com/emqx/emqx/pull/12802) Improved how EMQX handles node removal from clusters via the `emqx ctl cluster leave` command. Previously, nodes could unintentionally rejoin the same cluster (unless it was stopped) if the configured cluster `discovery_strategy` was not `manual`. With the latest update, executing the `cluster leave` command now automatically disables cluster discovery for the node, preventing it from rejoining. To re-enable cluster discovery, use the `emqx ctl discovery enable` command or simply restart the node. - [#​12814](https://redirect.github.com/emqx/emqx/pull/12814) Improved error handling for the `/clients/{clientid}/mqueue_messages` and `/clients/{clientid}/inflight_messages` APIs in EMQX. These updates address: - **Internal Timeout**: If EMQX fails to retrieve the list of Inflight or Mqueue messages within the default 5-second timeout, likely under heavy system load, the API will return 500 error with the response `{"code":"INTERNAL_ERROR","message":"timeout"}`, and log additional details for troubleshooting. - **Client Shutdown**: Should the client connection be terminated during an API call, the API now returns a 404 error, with the response `{"code": "CLIENT_SHUTDOWN", "message": "Client connection has been shutdown"}`. This ensures clearer feedback when client connections are interrupted. - [#​12824](https://redirect.github.com/emqx/emqx/pull/12824) Updated the statistics metrics `subscribers.count` and `subscribers.max` to include shared subscribers. Previously, these metrics accounted only for non-shared subscribers. - [#​12826](https://redirect.github.com/emqx/emqx/pull/12826) Fixed issues related to the import functionality of source data integrations and retained messages in EMQX. Before this update: - The data integration sources specified in backup files were not being imported. This included configurations under the `sources.mqtt` category with specific connectors and parameters such as QoS and topics. - Importing the `mnesia` table for retained messages was not supported. - [#​12843](https://redirect.github.com/emqx/emqx/pull/12843) Fixed `cluster_rpc_commit` transaction ID cleanup procedure on replicator nodes after executing the `emqx ctl cluster leave` command. Previously, failing to properly clear these transaction IDs impeded configuration updates on the core node. - [#​12885](https://redirect.github.com/emqx/emqx/pull/12885) Fixed an issue in EMQX where users were unable to view "Retained Messages" under the "Monitoring" menu in the Dashboard. The "Retained messages" backend API uses the `qlc` library. This problem was due to a permission issue where the `qlc` library's `file_sorter` function tried to use a non-writable directory, `/opt/emqx`, to store temporary files, resulting from recent changes in directory ownership permissions in Docker deployments. This update modifies the ownership settings of the `/opt/emqx` directory to `emqx:emqx`, ensuring that all necessary operations, including retained messages retrieval, can proceed without access errors.

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

This PR was generated by Mend Renovate. View the repository job log.