Import users' generated data from cloud watch to the Neon db

[kislerdm]

Problem

Users generated the data very valuable for model improvement. The data generated before 2023-02-16 are stored to cloudwatch instead of the core db.

Proposed Solution

• [x] Create the s3 bucket dev.diagramastext.data
• [x] Dump the cloudwatch the lambda core-c4 logs to the bucket with the prefix logsdump/lambda/
• [x] Parse the logs and extract the prompt and response - [ ] Import parsed data to the Neon database core on the basis of #41

Parsing details

• Use "00000000-0000-0000-0000-000000000000" as the placeholder for user_id value. The rational: the user is not set on the client side yet.
• Use lambda request_id as the value for the tables' field request_id.
• Database contract:
  • user_prompt DDL
  • openai_response DDL

Example

GIVEN

The log input:

2023-02-14T23:07:04.021Z INIT_START Runtime Version: go:1.v13   Runtime Version ARN: arn:aws:lambda:us-east-2::runtime:3f96445bbbd5587be5ace0de37b48e731074fa350870a8a3cb87f45011d70281

2023-02-14T23:07:04.105Z START RequestId: a54aa4b0-744d-4e1f-91d8-7eb26f99dd6f Version: $LATEST

2023-02-14T23:07:04.106Z 2023/02/14 23:07:04 {"prompt":"C4 diagram of a Go web server reading from external Postgres database over TCP without the big part"}

2023-02-14T23:07:05.576Z 2023/02/14 23:07:05 {"raw":"{"id":"cmpl-6jyg4CeNcXH7dDfOrbWFKYFuBIfVu","object":"text_completion","created":1676416024,"model":"code-cushman-001","choices":[{"text":"{\"nodes\":[{\"id\":\"0\",\"label\":\"Web Server\",\"technology\":\"Go\",\"description\":\"Authenticates users\"},{\"id\":\"1\",\"label\":\"Database\",\"technology\":\"Postgres\",\"external\":true,\"is_database\":true}],\"links\":[{\"from\":\"0\",\"to\":\"1\",\"label\":\"reads from database\",\"technology\":\"TCP\"}]}","index":0,"logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":842,"completion_tokens":210,"total_tokens":1052}}

2023-02-14T23:07:05.576Z "}

2023-02-14T23:07:05.576Z 2023/02/14 23:07:05 {"cleaned":"{"nodes":[{"id":"0","label":"Web Server","technology":"Go","description":"Authenticates users"},{"id":"1","label":"Database","technology":"Postgres","external":true,"is_database":true}],"links":[{"from":"0","to":"1","label":"reads from database","technology":"TCP"}]}"}

2023-02-14T23:07:06.987Z END RequestId: a54aa4b0-744d-4e1f-91d8-7eb26f99dd6f

2023-02-14T23:07:06.987Z REPORT RequestId: a54aa4b0-744d-4e1f-91d8-7eb26f99dd6f Duration: 2882.69 ms    Billed Duration: 2883 ms    Memory Size: 256 MB Max Memory Used: 39 MB  Init Duration: 82.83 ms

WHEN

Apply parsing logic - TBD in terms of the task

THEN

The extracted values are expected to correspond to the following fields of db tables.

user_prompt:	request_id	user_id	prompt	timestamp
a54aa4b0-744d-4e1f-91d8-7eb26f99dd6f	00000000-0000-0000-0000-000000000000	C4 diagram of a Go web server reading from external Postgres database over TCP without the big part	2023-02-14T23:07:04.106Z 2023/02/14 23:07:04

openai_response:	request_id	user_id	response	timestamp
a54aa4b0-744d-4e1f-91d8-7eb26f99dd6f	00000000-0000-0000-0000-000000000000	{"nodes":[{"id":"0","label":"Web Server","technology":"Go","description":"Authenticates users"},{"id":"1","label":"Database","technology":"Postgres","external":true,"is_database":true}],"links":[{"from":"0","to":"1","label":"reads from database","technology":"TCP"}]}	2023-02-14T23:07:05.576Z 2023/02/14 23:07:05

Acceptance Criteria

• All lambda logs until 2023-02-16 inclusive are parsed
• Parsing results are stored to the core database, tables user_prompt and openai_response

[kislerdm]

@ColeDrain Hey Ugo! Please find the raw logs data available here.

[ColeDrain]

@kislerdm Logs successfully parsed, the csv files are stored on the repo.

[kislerdm]

@ColeDrain thanks a lot for your contribution, Ugo! 🙏🏻