search

terascope / teraslice

Scalable data processing pipelines in JavaScript
https://terascope.github.io/teraslice/
Apache License 2.0
50 stars 13 forks source link

s3_exporter error - Failure to setup bucket - Teraslice 2.2.0 #3739

Closed godber closed 2 months ago

godber commented 2 months ago

Our internal testing of the Teraslice 2.2.0 release experienced issues with the s3_exporter,

the job is as follows:

{
    "name": "dg-to-s3-s3-3-0-0",
    "lifecycle": "once",
    "workers": 3,
    "operations": [
        {
            "_op": "data_generator",
            "size": 2000000
        },
        {
            "_op": "s3_exporter",
            "path": "/ts-v2-2-0-nodev18-20-4-1725407419656-test2-dg-to-s3",
            "format": "ldjson",
            "size": 10000,
            "connection": "tera4-minio"
        }
    ],
    "assets": [
        "file:3.0.0",
        "standard:1.0.1"
    ],
    "job_id": "4018757c-5862-4972-b78a-d6cf44d51de0",
    "_context": "job",
    "_created": "2024-09-03T23:50:23.916Z",
    "_updated": "2024-09-03T23:50:23.916Z"
}

Stack trace from the logs:

[2024-09-04T00:55:45.312Z] ERROR: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: Teraslice Worker shutting down due to failure! (assignment=worker, err.code=INTERNAL_SERVER_ERROR)
    TSError: Failure to setup bucket /ts-v2-2-0-nodev18-20-4-1725407419656-test2-dg-to-s3, caused by Error: getEndpointParameterInstructions() is not defined on `CreateBucketCommand`
        at S3Sender.ensureBucket (file:///app/assets/6cdfa4cad29caaad0dbe746150c224197a68550d/index.js:177050:15)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async S3SenderAPI.create (file:///app/assets/6cdfa4cad29caaad0dbe746150c224197a68550d/index.js:178052:5)
        at async Object.create (file:///app/assets/6cdfa4cad29caaad0dbe746150c224197a68550d/index.js:170540:36)
        at async S3Batcher.initialize (file:///app/assets/6cdfa4cad29caaad0dbe746150c224197a68550d/index.js:177819:16)
        at async file:///app/source/packages/job-components/dist/src/execution-context/base.js:67:17
        at async /app/source/node_modules/p-map/index.js:57:22

Full Logs from the worker:

[2024-09-04T00:55:42.471Z]  INFO: teraslice/19 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: (assignment=worker, module=asset_loader, worker_id=CG4kuXZk, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
    Loading assets... [
      '6cdfa4cad29caaad0dbe746150c224197a68550d',
      '5ab91d2dd9b17e742b652e7d8c4798c54a5b65b0'
    ]
[2024-09-04T00:55:42.473Z]  INFO: teraslice/19 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: creating connection for s3 (assignment=worker)
[2024-09-04T00:55:42.475Z]  INFO: teraslice/19 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: Using S3 endpoint: https://minio.XXXX (assignment=worker)
[2024-09-04T00:55:42.485Z]  INFO: teraslice/19 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: creating connection for elasticsearch-next (assignment=worker)
[2024-09-04T00:55:42.724Z]  INFO: teraslice/19 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: assets storage initialized (assignment=worker, module=assets_storage, worker_id=CG4kuXZk, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:42.914Z]  INFO: teraslice/19 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: loading assets: 6cdfa4cad29caaad0dbe746150c224197a68550d (assignment=worker, module=asset_loader, worker_id=CG4kuXZk, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:42.921Z]  INFO: teraslice/19 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: loading assets: 5ab91d2dd9b17e742b652e7d8c4798c54a5b65b0 (assignment=worker, module=asset_loader, worker_id=CG4kuXZk, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:42.921Z]  INFO: teraslice/19 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: decompressing and saving asset 6cdfa4cad29caaad0dbe746150c224197a68550d to /app/assets/6cdfa4cad29caaad0dbe746150c224197a68550d (assignment=worker, module=asset_loader, worker_id=CG4kuXZk, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:43.046Z]  INFO: teraslice/19 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: decompressed asset 6cdfa4cad29caaad0dbe746150c224197a68550d to /app/assets/6cdfa4cad29caaad0dbe746150c224197a68550d (assignment=worker, module=asset_loader, worker_id=CG4kuXZk, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:43.056Z]  INFO: teraslice/19 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: (assignment=worker, module=asset_loader, worker_id=CG4kuXZk, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
    asset 6cdfa4cad29caaad0dbe746150c224197a68550d saved to file /app/assets/6cdfa4cad29caaad0dbe746150c224197a68550d {
      id: '6cdfa4cad29caaad0dbe746150c224197a68550d',
      name: 'file',
      version: '3.0.0',
      description: 'A set of processors for working with files',
      node_version: 18
    }
[2024-09-04T00:55:43.057Z]  INFO: teraslice/19 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: decompressing and saving asset 5ab91d2dd9b17e742b652e7d8c4798c54a5b65b0 to /app/assets/5ab91d2dd9b17e742b652e7d8c4798c54a5b65b0 (assignment=worker, module=asset_loader, worker_id=CG4kuXZk, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:43.223Z]  INFO: teraslice/19 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: decompressed asset 5ab91d2dd9b17e742b652e7d8c4798c54a5b65b0 to /app/assets/5ab91d2dd9b17e742b652e7d8c4798c54a5b65b0 (assignment=worker, module=asset_loader, worker_id=CG4kuXZk, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:43.224Z]  INFO: teraslice/19 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: (assignment=worker, module=asset_loader, worker_id=CG4kuXZk, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
    asset 5ab91d2dd9b17e742b652e7d8c4798c54a5b65b0 saved to file /app/assets/5ab91d2dd9b17e742b652e7d8c4798c54a5b65b0 {
      id: '5ab91d2dd9b17e742b652e7d8c4798c54a5b65b0',
      name: 'standard',
      version: '1.0.1',
      description: 'Teraslice standard processor asset bundle',
      node_version: 18
    }
[2024-09-04T00:55:45.231Z]  WARN: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: Cannot create PromMetricsAPI because metrics are disabled. (assignment=worker, module=prom_metrics)
[2024-09-04T00:55:45.233Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: creating connection for elasticsearch-next (assignment=worker)
[2024-09-04T00:55:45.245Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: creating connection for elasticsearch-next (assignment=worker)
[2024-09-04T00:55:45.273Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: state storage initialized (assignment=worker, module=state_storage, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.274Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: analytics storage initialized (assignment=worker, module=analytics_storage, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.297Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [START] "s3_sender_api:s3_exporter-1" api instance initialize (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.298Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [FINISH] "s3_sender_api:s3_exporter-1" api instance initialize, took 1ms (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.298Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [START] "job-observer" operation initialize (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.298Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [START] "s3_sender_api:s3_exporter-1" operation initialize (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.298Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [START] "data_generator" operation initialize (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.298Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [START] "s3_exporter" operation initialize (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.298Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [FINISH] "job-observer" operation initialize, took 0ms (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.298Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [FINISH] "s3_sender_api:s3_exporter-1" operation initialize, took 0ms (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.298Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [FINISH] "data_generator" operation initialize, took 0ms (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.299Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: creating connection for s3 (assignment=worker)
[2024-09-04T00:55:45.299Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: Using S3 endpoint: https://minio-XXXXX (assignment=worker)
[2024-09-04T00:55:45.312Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [FINISH] "s3_exporter" operation initialize, took 14ms (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.312Z] ERROR: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: Teraslice Worker shutting down due to failure! (assignment=worker, err.code=INTERNAL_SERVER_ERROR)
    TSError: Failure to setup bucket /ts-v2-2-0-nodev18-20-4-1725407419656-test2-dg-to-s3, caused by Error: getEndpointParameterInstructions() is not defined on `CreateBucketCommand`
        at S3Sender.ensureBucket (file:///app/assets/6cdfa4cad29caaad0dbe746150c224197a68550d/index.js:177050:15)
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async S3SenderAPI.create (file:///app/assets/6cdfa4cad29caaad0dbe746150c224197a68550d/index.js:178052:5)
        at async Object.create (file:///app/assets/6cdfa4cad29caaad0dbe746150c224197a68550d/index.js:170540:36)
        at async S3Batcher.initialize (file:///app/assets/6cdfa4cad29caaad0dbe746150c224197a68550d/index.js:177819:16)
        at async file:///app/source/packages/job-components/dist/src/execution-context/base.js:67:17
        at async /app/source/node_modules/p-map/index.js:57:22
[2024-09-04T00:55:45.412Z]  WARN: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: worker shutdown was called for execution ac78ea3f-8127-433d-aec6-312fb3902b03 due to event: error (assignment=worker, module=worker, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.413Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [START] "job-observer" operation shutdown (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.413Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [START] "s3_sender_api:s3_exporter-1" operation shutdown (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.413Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [START] "data_generator" operation shutdown (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.413Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [START] "s3_exporter" operation shutdown (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.413Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [FINISH] "job-observer" operation shutdown, took 0ms (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.413Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [FINISH] "s3_sender_api:s3_exporter-1" operation shutdown, took 0ms (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.413Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [FINISH] "data_generator" operation shutdown, took 0ms (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.413Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: [FINISH] "s3_exporter" operation shutdown, took 0ms (assignment=worker, module=worker_context, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.413Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: shutting down (assignment=worker, module=state_storage, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.414Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: shutting down. (assignment=worker, module=analytics_storage, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.416Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: client 10.32.29.161__9ALjZF0Y disconnected { reason: 'io client disconnect' } (assignment=worker, module=messaging:client, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.416Z]  WARN: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: worker 10.32.29.161__9ALjZF0Y is shutdown for execution ac78ea3f-8127-433d-aec6-312fb3902b03, processed 0 slices (assignment=worker, module=worker, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:45.416Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: worker shutdown, already shutting down, remaining 5m (assignment=worker, module=worker:shutdown_handler, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)
[2024-09-04T00:55:46.418Z]  INFO: teraslice/8 on ts-wkr-dg-to-s3-s3-3-0-0-4018757c-5862-5db7cc8c8c-cpcgd: worker shutdown took 1s, exit with 1 status code (assignment=worker, module=worker:shutdown_handler, worker_id=9ALjZF0Y, ex_id=ac78ea3f-8127-433d-aec6-312fb3902b03, job_id=4018757c-5862-4972-b78a-d6cf44d51de0)

Maybe this should be filed on the file_asset project.

godber commented 2 months ago

The bucket does not exist before the job starts. Perhaps I should have had the test repeated with the 3.0.1 file asset.

godber commented 2 months ago

There was a type error in the v3.0.0 file-assets that only manifests in Teraslice v2.2.0, this is resolved in the v3.0.1 release of the file-assets. All of the release have been updated with warnings.

Fixed in this mr: https://github.com/terascope/file-assets/pull/1048

Closing.