Handling unstructured maps

[raryanpur]

Been loving ElectroDB, thanks for working on it!

Is there a way to model unstructured/partially structured maps in an entity schema? The use case is for complex and potentially large json objects (usually coming from a third party service) stored as an entity's attribute where some subset of the data is relevant to the application interacting with DynamoDB, but the rest isn't.

For example, an attribute that holds data from a geomapping service wherein the geomapping record is large and hence tedious to model as a map schema by hand. In such a case, the application may be interested in a few properties like lat/lon, but the rest (e.g. feature points, descriptions, etc.) are only relevant to consumers (e.g. a front-end client).

What ElectroDB does (please correct if anything is wrong!)

ElectroDB doesn't allow omitting the properties property on schema attributes of type map. From what I've observed properties works as a whitelist, so the full schema of the map has to be defined otherwise data will be removed when persisted. This can be very tedious and clutter up schema definitions, especially when the application only needs to access certain data within the map.

Possible solutions

• Bite the bullet and model the json object fully. Not preferred for reasons mentioned.
• Serialize/deserialize json via set/get and change the attribute type to string. Should work in a pinch, but loses the ability to use filter expressions to drill into maps. Not preferred.
• Add a schemaless (or some other name) property to map attributes, which prevents properties from acting as a whitelist. So schema checks will be applied to whatever properties are specified, but the rest of the map is left alone and can be saved to Dynamo as-is.

Very possible I've missed something, so please let me know if any clarifications are needed!

[tywalch]

A quick flyby comment while I have a second, in case this might solve your need quickly: You can define a custom attribute (that's strongly typed) using this function: https://electrodb.dev/en/reference/typescript/#customattributetype

Does that get you where you need to be?

[raryanpur]

Thanks for the prompt reply! I'm actually working with JS without TS, so not sure whether custom attribute types apply?

If they do, just taking a look over the examples, something that jumped out was this comment,

For complex objects and arrays, the base object would be “any” but you can also use a base type like “string”, “number”, or “boolean”

If the base type is "any" and not "map", does that mean filter expressions on nested map properties wouldn't work for that attribute?

[tywalch]

If you're just using JavaScript, you can use the attribute type "any". Filtering with the "where" method will work on "any" attribute types.

[raryanpur]

If you're just using JavaScript, you can use the attribute type "any". Filtering with the "where" method will work on "any" attribute types.

Ah ok, makes sense! I see now that the AWS SDK DocumentClient is used under the hood for marshalling/unmarshalling between JS types and Dynamo. Was thinking that the attribute types in the schema definition were used but those are only for validations (makes sense).

I wonder if a feature like this would still be useful to take advantage of the map validation functionality that ElectroDB has built-in? Thinking perhaps conceptually it's more like a 'trim' option than 'schemaless' as I suggested before. In other words, trim: true would be the default (current) behavior, whereas trim: false would keep (but not validate) keys that are in the map but not specified in the map's schema properties.

Anyway, changing the type to 'any' and running custom validations on the attribute's structure/values unblocks me for now. Will do that, thanks!

[SamWSoftware]

I had the same 'issue' and custom attribute type worked brilliantly!

config: { type: CustomAttributeType<Record<string, { type: ConfigTypes }>>("any"), required: true },