Filtering of the log messages?

[epugh]

I notice the message that is logged when the blob is written to the DB is pretty massive... Is there any way to filter it out?

Here is what I see (with the data all removed except for the first six bytes...

ActiveStorageDB::File Create (2.8ms)  INSERT INTO `active_storage_db_files` (`ref`, `data`, `created_at`) VALUES ('7313t5zetar6ndasua7kpqb7q651', x'789ced

with Mysql, do we have any sense of how large you can make a file? I am storing large JSON files (like up to 100 mb) for background processing.. goign to play with compressing them before uploading!

[epugh]

I wanted to share that @reid-rigo made a TruncatingFormatter https://github.com/o19s/quepid/blob/main/lib/truncating_formatter.rb that is working great for this.

Really appreciate this project, it fills a maybe niche but super valuable need!

[blocknotes]

Hey @epugh, from my standpoint, logs are not a direct responsibility of the component. They should be handled in the app. The truncating formatter that you linked could be an option I suppose.

About the maximum file size of a binary blob: honestly I don't know, but it probably depends on the kind of DB server, its version, extra DB plugins, etc. 🤷‍♂️

[epugh]

I think the only reason I call out the log component is because when you add in this gem, all of sudden you are seeing really large logs! It make sense, we are using the db to store binary data... Just a "oh, how do I deal with that" kind of thing.

Regardless, the gem continues to work great for Quepid!

[blocknotes]

Mmm... I made some checks, locally in the dummy app (using Rails 7.0.8.1 + Postgres 12) the data content is replaced with ("<2848 bytes of binary data>") in the INSERT statement:

ActiveStorageDB::File Create (0.7ms)  INSERT INTO "active_storage_db_files" ("ref", "data", "created_at") VALUES ($1, $2, $3) RETURNING "id"  [["ref", "al7swifxou1z454utc82pyx2q1di"], ["data", "<2848 bytes of binary data>"], ["created_at", "2024-02-29 16:48:15.963008"]]

Checking the schema.rb the table is created with:

  create_table "active_storage_db_files", force: :cascade do |t|
    t.string "ref", null: false
    t.binary "data", null: false
    t.datetime "created_at", null: false
    t.index ["ref"], name: "index_active_storage_db_files_on_ref", unique: true
  end

If I understood correctly, it should be filtered out because of the binary type for data. Perhaps what you report is system-specific 🤔

[blocknotes]

Another point that you could evaluate is this Rails PR: https://github.com/rails/rails/pull/42006 It looks like that you could filter specific columns in the query logs. But honestly I'm not sure if it works, it could be for SELECT queries only 🤷‍♂️

[billybonks]

That PR only filters from prepared statements; I am not sure why such an arbitrary choice was made