Buffer fills up while using Node.js/Lambda

[joshk132]

Steps to reproduce:

1. Setup InfluxDB Cloud free tier
2. Write to DB
3. Check logs

'use strict';

const {InfluxDB, Point, HttpError} = require('@influxdata/influxdb-client')

const InfluxURL = 'https://us-west-2-1.aws.cloud2.influxdata.com'
const token = 'mytoken'
const org = 'myorg'
const bucket= 'mybucket'

const writeApi = new InfluxDB({url: InfluxURL, token}).getWriteApi(org, bucket, 'ms')

module.exports.perf = async (event, context, callback) => {
  context.callbackWaitsForEmptyEventLoop = false;

  let input = JSON.parse(event.Records[0].body);
  console.log(input)

  const point1 = new Point('elapsedTime')
    .tag(input.foo, 'foo')
    .floatField('value', input.bar)
  writeApi.writePoint(point1)
  writeApi
  .close()
  .then((result) => {
    console.log(result)
  })
  .catch(e => {
    console.error(e)
    if (e instanceof HttpError && e.statusCode === 401) {

    }
    console.log('\nFinished ERROR')
  })

  return true
};

Expected behavior: I expect it to write every data point to the DB for each SQS message - takes one message with one data point per lambda execution.

Actual behavior: It fills the buffer - which IDK how to clear - and prevents any data from being written to the DB

Environment info: Cloud offering free tier on AWS us-west-2

Logs: ERROR: Retry buffer closed with 1 items that were not written to InfluxDB!

Notes: I did manage to write one data point a few days ago and since then it has been not letting me write anything saying the buffer is full

[russorat]

@joshk132 thanks! Based on this example: https://github.com/influxdata/influxdb-client-js/blob/master/examples/write.js#L30

It looks like you can call writeApi.flush() after each SQS message if you'd like to write after each point. Also, i'm not a lambda expert, but if you are closing the connection after each function call, I think you will need to open it inside the function call as well. If you change it to use flush, you might be ok.

[ZaneL]

What is the solution for this? I'm encountering the same problem...

writeApi.flush() doesn't fix it.

[sranka]

@joshk132 @ZaneL

• writeApi is closed from a function that is not creating it, this might cause problems; A closed WriteApi will not write points to In fluxDB. The next influxdb-client-js release will report an error when using writePoint with a closed WriteApi ( #246 )
• the perf function is async, therefore you should better await writeApi.close, and avoid promise's then/catch fn

Try this:

module.exports.perf = async (event, context, callback) => {
  context.callbackWaitsForEmptyEventLoop = false;
  const writeApi = new InfluxDB({url: InfluxURL, token}).getWriteApi(org, bucket, 'ms')

  let input = JSON.parse(event.Records[0].body);
  console.log(input)

  const point1 = new Point('elapsedTime')
    .tag(input.foo, 'foo')
    .floatField('value', input.bar)
  try {
    writeApi.writePoint(point1)
    const result = await writeApi.close()
    console.log(result)
  } catch(e){
    console.error(e)
    console.log('\nFinished ERROR')
  })

  return true
};

Please see also https://community.influxdata.com/t/problem-writing-to-influxdb-cloud-from-aws-lambda/15705/3 , it might be relevant as well.

[ZaneL]

Thank you!

That was the problem -- looks like I just needed to await on writeApi.close()