Snyk has created this PR to upgrade @cumulus/logger from 18.3.0 to 18.3.1.
:information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project.
The recommended version is 1 version ahead of your current version.
The recommended version was released on 21 days ago.
The following applies only to users with a custom value configured for async_operation_image:
As part of the node v20 update process, a new version (52) of the Core
async-operation container was published - cumuluss/async
operation The
default value for async_operation_image has been updated in the cumulus
module, however if you are using an internal image repository such as ECR,
please make sure to update your deployment configuration with the newly
provided image.
Users making use of a custom image configuration should note the base image
for Core async operations must support node v20.x.
CUMULUS-3617 Migration of DLA messages should be performed after Cumulus is upgraded
Instructions for migrating old DLA (Dead Letter Archive) messages to new format:
YYYY-MM-DD subfolders to organize by date
new top level fields for simplified search and analysis
captured error message
To invoke the Lambda and start the DLA migration, you can use the AWS Console or CLI:
OUTFILE (optional) is the filepath where the Lambda output will be saved.
The Lambda will trigger an Async Operation and return an id such as:
{"id":"41c9fbbf-a031-4dd8-91cc-8ec2d8b5e31a","description":"Migrate Dead Letter Archive Messages",
"operationType":"DLA Migration","status":"RUNNING",
"taskArn":"arn:aws:ecs:us-east-1:AWSID:task/$PREFIX-CumulusECSCluster/123456789"}
which you can then query the Async Operations API
Endpoint for the
output or status of your request. If you want to directly observe the progress
of the migration as it runs, you can view the CloudWatch logs for your async
operations (e.g. PREFIX-AsyncOperationEcsLogs).
CUMULUS-3779 async_operations Docker image version upgrade
The async-operation Docker image has been updated to support Node v20 and aws-sdk v3. Users of the image will need
to update to at least async-operations:52.
CUMULUS-3776 cumulus-ecs-task Docker image version upgrade
The cumulus-ecs-task Docker image has been updated to support Node v20 and aws-sdk v3. Users of the image will need
to update to at least cumulus-ecs-task:2.1.0.
Breaking Changes
CUMULUS-3618
Modified @ cumulus/es-client/search.BaseSearch:
Removed static class method es in favor of new class for managing
elasticsearch clients EsClient which allows for credential
refresh/reset. Updated api/es-client code to
utilize new pattern. Users making use of @ cumulus/es-client should
update their code to make use of the new EsClient create/initialize pattern.
Added helper method getEsClient to encapsulate logic to create/initialize
a new EsClient.
CUMULUS-2889
Removed unused CloudWatch Logs AWS SDK client. This change removes the CloudWatch Logs
client from the @ cumulus/aws-client package.
CUMULUS-2890
Removed unused CloudWatch AWS SDK client. This change removes the CloudWatch client
from the @ cumulus/aws-client package.
CUMULUS-3323
Updated @ cumulus/db to by default set the ssl option for knex, and
reject non-SSL connections via use of the rejectUnauthorized configuration
flag. This causes all Cumulus database connections to require SSL (CA or
self-signed) and reject connectivity if the database does not provide SSL.
Users using serverless v1/cumulus-rds-tf should not be impacted by this
change as certs are provided by default. Users using databases that do not
provide SSL should update their database secret with the optional value disableSSL set to true
Updated cumulus-rds-tf to set rds.force_ssl to 1, forcing SSL enabled
connections in the db_parameters configuration. Users of this module
defining their own db_parameters should make this configuration change to allow only SSL
connections to the RDS datastore.
CUMULUS-2897
Removed unused Systems Manager AWS SDK client. This change removes the Systems Manager client
from the @ cumulus/aws-client package.
CUMULUS-3779
Updates async_operations Docker image to Node v20 and bumps its cumulus dependencies to v18.3.0 to
support aws-sdk v3 changes.
Added
CUMULUS-3614
tf-modules/monitoring module now deploys Glue table for querying dead-letter-archive messages.
CUMULUS-3616
Added user guide on querying dead-letter-archive messages using AWS Athena.
CUMULUS-3433
Added importGot helper method to import got as an ESM module in
CommmonJS typescript/webpack clients.
CUMULUS-3606
Updated with additional documentation covering tunneling configuration
using a PKCS11 provider
Changed
CUMULUS-3735
Remove unused getGranuleIdsForPayload from @ cumulus/api/lib
CUMULUS-3746
cicd unit test error log changed to environment unique name
CUMULUS-3717
Update @ cumulus/ingest/HttpProviderClient to use direct injection test mocks, and remove rewire from unit tests
CUMULUS-3720
add cicd unit test error logging to s3 for testing improvements
CUMULUS-3433
Updated all node.js lambda dependencies to node 20.x/20.12.2
Modified @ cumulus/ingest unit test HTTPs server to accept localhost POST
requests, and removed nock dependency from tests involving fs.Readstream
and got due to a likely incompatibility with changes in node v18, got,
fs.Readstream and nock when used in combination in units
(sindresorhus/got#2341)
Updated got dependency in @ cumulus/ingest to use @ cumulus/common
dynamic import helper / got > v10 in CommonJS.
dla guarantees de-nested SQS message bodies, preferring outermost metadata as found.
dla uses execution Name as filename and ensures no ':' or '/' characters in name
CUMULUS-3570
Updated Kinesis docs to support latest AWS UI and recommend server-side encryption.
CUMULUS-3519
Updates SQS and SNS code to AWS SDK V3 Syntax
CUMULUS-3609
Adds dla-migration lambda to async-operations to be used for updating existing DLA records
Moved hoistCumulusMessageDetails function from write-db-dlq-records-to-s3 lambda to @ cumulus/message/DeadLetterMessage
CUMULUS-3613
Updated writeDbRecordsDLQtoS3 lambda to write messages to YYYY-MM-DD subfolder of S3 dead letter archive.
CUMULUS-3518
Update existing usage of @ cumulus/aws-client lambda service to use AWS SDK v3 send syntax
Update Discover Granules lambda default memory to 1024 MB
CUMULUS-3600
Update docs to clarify CloudFront HTTPS DIT requirements.
CUMULUS-2892
Updates aws-client's EC2 client to use AWS SDK v3.
CUMULUS-2896
Updated Secrets Manager code to AWS SDK v3.
CUMULUS-2901
Updated STS code to AWS SDK v3.
CUMULUS-2898
Update Step Functions code to AWS SDK v3
CUMULUS-2902
Removes aws-sdk from es-client package by replacing credential fetching with
the @ aws-sdk/credential-providers AWS SDK v3 package.
Removes aws-sdk from all cumulus packages and replaces usages with AWS SDK v3 clients.
CUMULUS-3456
Added stateMachineArn, executionArn, collectionId, providerId, granules, status, time, and error fields to Dead Letter Archive message
Added cumulusError field to records in sfEventSqsToDbRecordsDeadLetterQueue
CUMULUS-3323
Added disableSSL as a valid database secret key - setting this in your database credentials will
disable SSL for all Core database connection attempts.
Added rejectUnauthorized as a valid database secret key - setting
this to false in your database credentials will allow self-signed certs/certs with an unrecognized authority.
Updated the default parameter group for cumulus-rds-tf to set force_ssl
to 1. This setting for the Aurora Serverless v1 database disallows non-SSL
connections to the database, and is intended to help enforce security
compliance rules. This update can be opted-out by supplying a non-default db_parameters set in the terraform configuration.
CUMULUS-3425
Update @ cumulus/lzards-backup task to either respect the lzards_provider
terraform configuration value or utilize lzardsProvider as part of the task
workflow configuration
Minor refactor of @ cumulus/lzards-api-client to:
Use proper ECMAScript import for @ cumulus/launchpad-auth
Update incorrect docstring
CUMULUS-3497
Updated example/cumulus-tf/orca.tf to use v9.0.4
CUMULUS-3610
Updated aws-client's ES client to use AWS SDK v3.
CUMULUS-3617
Added lambdas to migrate DLA messages to YYYY-MM-DD subfolder
Updated @ cumulus/aws-client/S3/recursivelyDeleteS3Bucket to handle bucket with more than 1000 objects.
CUMULUS-2891
Updated ECS code to aws sdk v3
Fixed
CUMULUS-3715
Update ProvisionUserDatabase lambda to correctly pass in knex/node debug
flags to knex custom code
CUMULUS-3721
Update lambda:GetFunctionConfiguration policy statement to fix error related to resource naming
CUMULUS-3701
Updated @ cumulus/api to no longer improperly pass PATCH/PUT null values to Eventbridge rules
CUMULUS-3618
Fixed @ cumulus/es-client credentialing issue in instance where
lambda/Fargate task runtime would exceed the timeout for the es-client. Added retry/credential
refresh behavior to @ cumulus/es-client/indexer.genericRecordUpdate to ensure record indexing
does not fail in those instances.
Updated index-from-database lambda to utilize updated es-client to prevent
credentialing timeout in long-running ECS jobs.
CUMULUS-3323
Minor edits to errant integration test titles (dyanmo->postgres)
AWS-SDK v3 Exclusion (v18.3.0 fix)*
Excludes aws-sdk v3 from packages to reduce overall package size. With the requirement of Node v20
packaging the aws-sdk v3 with our code is no longer necessary and prevented some packages from being
published to npm.
Snyk has created this PR to upgrade @cumulus/logger from 18.3.0 to 18.3.1.
:information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project.
The recommended version is 1 version ahead of your current version.
The recommended version was released on 21 days ago.
Release notes
Package name: @cumulus/logger
-
18.3.1 - 2024-07-09
-
- new top level fields for simplified search and analysis
- captured error message
-
- Modified @ cumulus/es-client/search.BaseSearch:
- Removed static class method
- Added helper method getEsClient to encapsulate logic to create/initialize
-
- Removed unused CloudWatch Logs AWS SDK client. This change removes the CloudWatch Logs
-
- Removed unused CloudWatch AWS SDK client. This change removes the CloudWatch client
-
- Updated
- Updated
-
- Removed unused Systems Manager AWS SDK client. This change removes the Systems Manager client
-
- Updates async_operations Docker image to Node v20 and bumps its cumulus dependencies to v18.3.0 to
- CUMULUS-3614
- CUMULUS-3616
- Added user guide on querying dead-letter-archive messages using AWS Athena.
- CUMULUS-3433
- Added
- CUMULUS-3606
- Updated with additional documentation covering tunneling configuration
- CUMULUS-3735
- Remove unused getGranuleIdsForPayload from
- CUMULUS-3746
- cicd unit test error log changed to environment unique name
- CUMULUS-3717
- Update
- CUMULUS-3720
- add cicd unit test error logging to s3 for testing improvements
- CUMULUS-3433
- Updated all node.js lambda dependencies to node 20.x/20.12.2
- Modified
- Updated
- Updated all Core lambdas to use cumulus-message-adapter-js v2.2.0
- CUMULUS-3629
- dla guarantees de-nested SQS message bodies, preferring outermost metadata as found.
- dla uses execution Name as filename and ensures no ':' or '/' characters in name
- CUMULUS-3570
- Updated Kinesis docs to support latest AWS UI and recommend server-side encryption.
- CUMULUS-3519
- Updates SQS and SNS code to AWS SDK V3 Syntax
- CUMULUS-3609
- Adds dla-migration lambda to async-operations to be used for updating existing DLA records
- Moved hoistCumulusMessageDetails function from write-db-dlq-records-to-s3 lambda to @ cumulus/message/DeadLetterMessage
- CUMULUS-3613
- Updated writeDbRecordsDLQtoS3 lambda to write messages to
- CUMULUS-3518
- Update existing usage of
- Update Discover Granules lambda default memory to 1024 MB
- CUMULUS-3600
- Update docs to clarify CloudFront HTTPS DIT requirements.
- CUMULUS-2892
- Updates
- CUMULUS-2896
- Updated Secrets Manager code to AWS SDK v3.
- CUMULUS-2901
- Updated STS code to AWS SDK v3.
- CUMULUS-2898
- Update Step Functions code to AWS SDK v3
- CUMULUS-2902
- Removes
- Removes
- CUMULUS-3456
- Added stateMachineArn, executionArn, collectionId, providerId, granules, status, time, and error fields to Dead Letter Archive message
- Added cumulusError field to records in sfEventSqsToDbRecordsDeadLetterQueue
- CUMULUS-3323
- Added
- Added
- Updated the default parameter group for
- CUMULUS-3425
- Update
- Minor refactor of
- Use proper ECMAScript import for
- Update incorrect docstring
- CUMULUS-3497
- Updated
- CUMULUS-3610
- Updated
- CUMULUS-3617
- Added lambdas to migrate DLA messages to
- Updated
- CUMULUS-2891
- Updated ECS code to aws sdk v3
- CUMULUS-3715
- Update
- CUMULUS-3721
- Update lambda:GetFunctionConfiguration policy statement to fix error related to resource naming
- CUMULUS-3701
- Updated
- CUMULUS-3618
- Fixed
- Updated
- CUMULUS-3323
- Minor edits to errant integration test titles (dyanmo->postgres)
- AWS-SDK v3 Exclusion (v18.3.0 fix)*
- Excludes aws-sdk v3 from packages to reduce overall package size. With the requirement of Node v20
-
18.3.0 - 2024-06-15
from @cumulus/logger GitHub release notesMigration Notes
CUMULUS-3433 Update to node.js v20
The following applies only to users with a custom value configured for
async_operation_image:As part of the node v20 update process, a new version (52) of the Core
async-operation container was published - cumuluss/async
operation The
default value for
async_operation_imagehas been updated in thecumulusmodule, however if you are using an internal image repository such as ECR,
please make sure to update your deployment configuration with the newly
provided image.
Users making use of a custom image configuration should note the base image
for Core async operations must support node v20.x.
CUMULUS-3617 Migration of DLA messages should be performed after Cumulus is upgraded
Instructions for migrating old DLA (Dead Letter Archive) messages to new format:
YYYY-MM-DDsubfolders to organize by dateTo invoke the Lambda and start the DLA migration, you can use the AWS Console or CLI:
PREFIXis your Cumulus deployment prefix.OUTFILE(optional) is the filepath where the Lambda output will be saved.The Lambda will trigger an Async Operation and return an
idsuch as:{"id":"41c9fbbf-a031-4dd8-91cc-8ec2d8b5e31a","description":"Migrate Dead Letter Archive Messages", "operationType":"DLA Migration","status":"RUNNING", "taskArn":"arn:aws:ecs:us-east-1:AWSID:task/$PREFIX-CumulusECSCluster/123456789"}which you can then query the Async Operations API
Endpoint for the
output or status of your request. If you want to directly observe the progress
of the migration as it runs, you can view the CloudWatch logs for your async
operations (e.g.
PREFIX-AsyncOperationEcsLogs).CUMULUS-3779 async_operations Docker image version upgrade
The
async-operationDocker image has been updated to support Node v20 andaws-sdkv3. Users of the image will needto update to at least async-operations:52.
CUMULUS-3776 cumulus-ecs-task Docker image version upgrade
The
cumulus-ecs-taskDocker image has been updated to support Node v20 andaws-sdkv3. Users of the image will needto update to at least cumulus-ecs-task:2.1.0.
Breaking Changes
CUMULUS-3618
esin favor of new class for managingelasticsearch clients
EsClientwhich allows for credentialrefresh/reset. Updated api/es-client code to
utilize new pattern. Users making use of @ cumulus/es-client should
update their code to make use of the new EsClient create/initialize pattern.
a new EsClient.
CUMULUS-2889
client from the
@ cumulus/aws-clientpackage.CUMULUS-2890
from the
@ cumulus/aws-clientpackage.CUMULUS-3323
@ cumulus/dbto by default set thessloption for knex, andreject non-SSL connections via use of the
rejectUnauthorizedconfigurationflag. This causes all Cumulus database connections to require SSL (CA or
self-signed) and reject connectivity if the database does not provide SSL.
Users using serverless v1/
cumulus-rds-tfshould not be impacted by thischange as certs are provided by default. Users using databases that do not
provide SSL should update their database secret with the optional value
disableSSLset totruecumulus-rds-tfto setrds.force_sslto1, forcing SSL enabledconnections in the
db_parametersconfiguration. Users of this moduledefining their own
db_parametersshould make this configuration change to allow only SSLconnections to the RDS datastore.
CUMULUS-2897
from the
@ cumulus/aws-clientpackage.CUMULUS-3779
support
aws-sdkv3 changes.Added
tf-modules/monitoringmodule now deploys Glue table for querying dead-letter-archive messages.importGothelper method to importgotas an ESM module inCommmonJS typescript/webpack clients.
using a PKCS11 provider
Changed
@ cumulus/api/lib@ cumulus/ingest/HttpProviderClientto use direct injection test mocks, and remove rewire from unit tests@ cumulus/ingestunit test HTTPs server to accept localhost POSTrequests, and removed nock dependency from tests involving
fs.Readstreamand
gotdue to a likely incompatibility with changes in node v18,got,fs.Readstream and nock when used in combination in units
(sindresorhus/got#2341)
gotdependency in@ cumulus/ingestto use@ cumulus/commondynamic import helper /
got> v10 in CommonJS.YYYY-MM-DDsubfolder of S3 dead letter archive.@ cumulus/aws-clientlambda service to use AWS SDK v3sendsyntaxaws-client's EC2 client to use AWS SDK v3.aws-sdkfromes-clientpackage by replacing credential fetching withthe
@ aws-sdk/credential-providersAWS SDK v3 package.aws-sdkfrom all cumulus packages and replaces usages with AWS SDK v3 clients.disableSSLas a valid database secret key - setting this in your database credentials willdisable SSL for all Core database connection attempts.
rejectUnauthorizedas a valid database secret key - settingthis to
falsein your database credentials will allow self-signed certs/certs with an unrecognized authority.cumulus-rds-tfto setforce_sslto 1. This setting for the Aurora Serverless v1 database disallows non-SSL
connections to the database, and is intended to help enforce security
compliance rules. This update can be opted-out by supplying a non-default
db_parametersset in the terraform configuration.@ cumulus/lzards-backuptask to either respect thelzards_providerterraform configuration value or utilize
lzardsProvideras part of the taskworkflow configuration
@ cumulus/lzards-api-clientto:@ cumulus/launchpad-authexample/cumulus-tf/orca.tfto use v9.0.4aws-client's ES client to use AWS SDK v3.YYYY-MM-DDsubfolder@ cumulus/aws-client/S3/recursivelyDeleteS3Bucketto handle bucket with more than 1000 objects.Fixed
ProvisionUserDatabaselambda to correctly pass in knex/node debugflags to knex custom code
@ cumulus/apito no longer improperly pass PATCH/PUT null values to Eventbridge rules@ cumulus/es-clientcredentialing issue in instance wherelambda/Fargate task runtime would exceed the timeout for the es-client. Added retry/credential
refresh behavior to
@ cumulus/es-client/indexer.genericRecordUpdateto ensure record indexingdoes not fail in those instances.
index-from-databaselambda to utilize updated es-client to preventcredentialing timeout in long-running ECS jobs.
packaging the aws-sdk v3 with our code is no longer necessary and prevented some packages from being
published to npm.
Release 18.3.0
Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.
For more information: