Array of Maps Schema Example

[pauhlbarnes]

[READ] Step 1: Are you in the right place?

Issues filed here should be about a feature request for a specific extension in this repository. To file a feature request that affects multiple extensions or the Firebase Extensions platform, please reach out to Firebase support directly.

[REQUIRED] Step 2: Extension name

This feature request is for extension: _ (firestore-bigquery-export)

What feature would you like to see?

Please provide a schema example for an array of Maps in the GENERATE_SCHEMA_VIEWS.md

How would you use it?

To enable breaking out of data stored as an array of maps.

[dackers86]

Hi @pauhlbarnes.

Given the current documentation, i'm not too sure if this something that is available in the schema generation.

Could you provide an example of the data structure you would intend to use, we have an array example, but i'm assuming the following does not apply?

{
    "fields": [
      {
        "name": "name",
        "type": "string"
      }
    ],
    "name": "friends",
    "type": "map"
 }

[pauhlbarnes]

Thanks, my data structure in firestore looks like the attached image

On Mon, Jan 25, 2021 at 2:06 PM Darren Ackers notifications@github.com wrote:

Hi @pauhlbarnes https://github.com/pauhlbarnes.

Given the current documentation, i'm not too sure if this something that is available in the schema generation.

Could you provide an example of the data structure you would intend to use, we have an array example, but i'm assuming the following does not apply?

{ "fields": [ { "name": "name", "type": "string" } ], "name": "friends", "type": "map" }

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/firebase/extensions/issues/573#issuecomment-766839503, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADEJS4RBLSY4B73NZQIFTOLS3V3GJANCNFSM4WOOSIHQ .

[dackers86]

Hi @pauhlbarnes

Apologies, I cannot see an attachment in your previous message?

Was there an attachment added when the issue was created?

[hernantz]

Hi @pauhlbarnes and @dackers86 I'm facing a similar issue, given this JSON structure

{
    "user": "foo",
    "friends": [
            {
                  "firstName": "John",
                  "lastName": "Doe"
            },
            {
                  "firstName": "Jane",
                  "lastName": "Doe"
            } 
    ]
}

This schema representation won't work, right?

{
    "fields": [
       {
            "name": "user",
            "type": "string"
        },
        {
            "name": "friends",
            "type": "map",
            "mode": "repeated",
            "fields": [
                {
                  "name": "firstName",
                  "type": "string"
                }, 
                {
                  "name": "lastName",
                  "type": "string"
                }
            ]
        }
    ]
}

Or at least this is what I understand from this comment from https://github.com/firebase/extensions/issues/583. Although it would seem to be possible to implement, according to the BigQuery docs on nested structs.

[darnmason]

Any intentions to support this array of maps?

My use case is similar to @hernantz

[dackers86]

HI @darnmason

I've added this to our project board for further investigation.

[polenvinagre]

Hi! I have been using this extension for a year and it is a great tool. Good job! Now I have a new project that has an: ARRAY OF MAPS Could you give some updates about this enhancement? Thanks in advance!

[Mr-Goldberg]

would be great if you implement this feature!

[zallesov]

Came here with the same request. The current implementation of an array is hazardous. A nested array has multiple entries which will result in multiple copies of a hosted object in BQ. Counting them will yield the wrong results.

[GaL3n]

The same thing for me, I have orders in WooCommerce format in a collection, and I can't analyze the meta_data within the line_items :(.

Exporting the entire collection manually, without using the extension, and importing it into Bigquery.. works.

[dackers86]

Hi All.

The main discussion for this feature, was include the PR we have recently merged.

This can be found here.

This update will be released soon.

As this is a tricky issue as there are so many ways this can be improved.

Please review the PR, if this does provide the solution you are looking for, we're happy to continue the discussion here and further enhance this feature.

[eigidi]

the feature supports typed arrays, but it does not seem to work for arrays of maps, like in this example:

{
  "fields": [
    {
      "name": "nickname",
      "type": "string"
    },
    {
      "name": "location",
      "type": "geopoint"
    },
    {
      "name": "profile",
      "type": "map",
      "fields": [
        {
          "name": "birthDate",
          "type": "timestamp"
        },
        {
          "name": "hand",
          "type": "string"
        },
        {
          "name": "levels",
          "type": "array",
          "fields": [
            {
              "name": "date",
              "type": "timestamp",
              "column_name": "levelDate"
            },
            {
              "name": "value",
              "type": "number",
              "column_name": "levelValue"
            }
          ]
        }
      ]
    }
  ]
}

[madmacc]

Any updates on this? Have just run into the same issue.

[pr-Mais]

Hello, since this issue is closed, could you please @eigidi open a new issue with the details of your request to better track it, thank you!

[eigidi]

I have submitted this issue 1963 with further details