flamelink sdk serverside issue - populate, fields

[shaunnez]

Here is how I setup my flamelink

// firebase init
import * as serviceAccount from './serviceAccountKey.json';
// @ts-ignore
const adminConfig = JSON.parse(process.env.FIREBASE_CONFIG);
// @ts-ignore
adminConfig.credential = admin.credential.cert(serviceAccount);
admin.initializeApp(adminConfig);
flamelinkApp(admin.app());

Requesting a collection flamelinkApp.content.get({ schemaKey: "foo", populate: false })

I would expect this to just give me back the "foo" collection with each field. It should not give me each relational field fully expanded as though "populate" is true.

I'm also getting back the "_firestore" object with my firestore credentials in each reference object which is nasty. E.g. Foo collection has "products".

{
    "products": [
                "product": {
                    "_firestore": {
                        "_settings": {
                            "credentials": {
                                "private_key": "-----BEGIN PRIVATE KEY-----REMOVED==\n-----END PRIVATE KEY-----\n",
                                "client_email": "REMOVED"
                            },
                            "projectId": "REMOVED,
                            "firebaseVersion": "7.0.0",
                            "libName": "gccl",
                            "libVersion": "1.3.0 fire/7.0.0"
                        },
                        "_settingsFrozen": true,
                        "_serializer": {
                            "timestampsInSnapshots": true
                        },
                        "_projectId": "REMOVED",
                        "_lastSuccessfulRequest": 1558669076626,
                        "_preferTransactions": false,
                        "_clientPool": {
                            "concurrentOperationLimit": 100,
                            "activeClients": {}
                        }
                    },
                    "_path": {
                        "segments": [
                            "fl_content",
                            "PKtzUU51iCtHbbdLXCab"
                        ]
                    }
                },
                "amount": 15,
                "uniqueKey": "keMllaEmX"
            }
  ]

So the only way I've been able to limit the collection to the data I want is to use "fields" as follows (assume foo has a relationship with user. flamelinkApp.content.get({ schemaKey: "foo", fields: [ "id", "name", "user.email"] })

However, this is taking 5 seconds to return 1 "foo"... Not sure why it's so slow.

Finally, how do I filter on a "repeater". I.e. if "foo" has "products"

I've tried all kinds of variations of fields and populate but no go.

Is it possible to use "fields" in the same way we do "populate"? E.g.

fields: [
 {
   field: "id".
 },
 {
   field: "name".
 },
 { 
   field: "products",
   fields: ["product.amount", "products.id"]
}

Thanks

[jperasmus]

Hi @shaunnez

Just before I try and look at the rest of your query, I just want to make sure the initialization of the flamelink app instance is 100%.

I assume you posted a simplified code snippet because it does not look 100% correct to me. Do you mind posting the full code snippet for how you initialize the Firebase admin instance as well as the Flamelink app instance?

Also, what version of the Flamelink SDK are you using?

[shaunnez]

Versions "firebase-admin": "~7.0.0", "firebase-functions": "^2.2.0", "flamelink": "^1.0.0-alpha.19",

Initialization

// firebase
import * as admin from 'firebase-admin';
import * as flamelink from 'flamelink/app';
// firebase init
import * as serviceAccount from './serviceAccountKey.json';
// @ts-ignore
const adminConfig = JSON.parse(process.env.FIREBASE_CONFIG);
// @ts-ignore
adminConfig.credential = admin.credential.cert(serviceAccount);
const fbApp = admin.initializeApp(adminConfig);
const flApp = flamelink({ firebaseApp: fbApp, dbType: 'cf' });

[shaunnez]

Heres an example of my firebase function / express route and an endpoint you can run a get on

GET: https://spotme-232200.firebaseapp.com/api/v1/quote

import * as express from 'express';
import * as admin from 'firebase-admin';
import * as flamelink from 'flamelink/app';
import * as serviceAccount from './serviceAccountKey.json';

// @ts-ignore
const adminConfig = JSON.parse(process.env.FIREBASE_CONFIG);
// @ts-ignore
adminConfig.credential = admin.credential.cert(serviceAccount);
const fbApp = admin.initializeApp(adminConfig);
const flApp = flamelink({ firebaseApp: fbApp, dbType: 'cf' });

const router = express.Router();
const collectionName = 'quote';
type collectionType = FSCollections.Quote;

router.get('/', async (req, res) => {
  try {
    const fields = ['id', 'quoteReference', 'user.email'];
    const populate = true;
    const data = await flApp.content.get({ schemaKey: collectionName, fields, populate });
    const arrData = [] as collectionType[];
    for (const k in data) {
      arrData.push(data[k]);
    }
    console.log(arrData);
    res.status(200).json(arrData);
  } catch (ex) {
    console.log(ex);
    res.status(404);
  }
});

export default router;

[jperasmus]

Thanks for all the info 👍

One thing I can see is where you import the Flamelink SDK, you correctly import the app (flamelink/app) module, but I do not see the other individual modules being imported. You will also need to import those after the app module:

import * as flamelink from 'flamelink/app';
import 'flamelink/content';
import 'flamelink/storage';
// etc

The second part is about the references to populate: for the Real-time Database, we store only the reference ID's and then populate those into the full objects. For Cloud Firestore, they have the concept of storing reference types as fields in your database, so we are using those. You can use and interact with these references like you would normally do within your app queries. You are most likely seeing these references logged out instead of the fields being populated in one of your first examples.

[shaunnez]

No problem. So for typescript your documentation says

import * as flamelink from 'flamelink/app' import 'flamelink/settings' import 'flamelink/content' import 'flamelink/storage' import 'flamelink/navigation' import 'flamelink/users'

I still get the following output in the console in my firebase functions app

[Flamelink] Warning!

Importing the whole Flamelink SDK is fine for development or quick prototyping,
but it is highly recommended that you only import the individual modules used
by your application when you use this SDK in production.

Instead of importing everything:
const flamelink = require('flamelink')

Only import the app and specific modules you use:
const flamelink = require('flamelink/app')
require('flamelink/<module>')

Where <module> can be one of 'content', 'navigation', 'storage', 'users' or 'settings'

Anyway regardless of the above.... My main question is about using fields and populate to retrieve the specific data I want from my collection with reference fields. For example:

quote -> 1-1 relationship with user quote -> repeater of "products" products -> contains an amount field and a relationship with "product" product -> id, title, description, and an repeater of "images" images -> title, description and a relationship with "image" image -> url object

quote { 
  id, 
  user {
    id,
    email
  },
  products: [
     {
        amount,
        product: {
            id,
            title,
            description,
            images: [
               {
                   title,
                   description,
                   image: [
                       {
                           url
                       }
                   ]
               }
            ]
        }
     }
  ]
}

My assumption (based on the documentation here) is that I should be using a combination of fields and populate (with fields and subfields) to specifically retrieve the data I want.

I've found that in the top level "fields" property I can specify the below that will retrieve me some of the information I want, but way to much information from products.

const fields = ['id', 'user.id', 'user.email', 'products']

I then tried the below and this gave me the product information I wanted but only for the first product.

const fields = ['id', 'user.id', 'user.email', 'products[0].amount', 'products[0].product.title']

So I need a way to specify just the fields I want on the array of products... E.g. products[*].product.title

I then tried a bunch of combinations of "fields" and "populate" and "subFields" but unless I specifically put in the fields I want in the "top level" fields array, I get back everything.

Ideally, I'd like to be able to specify this

    const fields = ['id', 'user', 'products'];
    const populate = [
      {
        field: 'user',
        fields: ['id', 'email']
      },
      {
        field: 'products',
        fields: ['amount', 'product'],
        subFields: [
          {
            field: 'product',
            fields: ['id', 'title', 'description', 'enabled']
          }
        ]
      }
    ];
    const data = await flApp.content.get({ schemaKey: collectionName, fields, populate });

Here is an image of the JSON response I'm getting once I convert it into an array.

You can view the actual API here

https://spotme-232200.firebaseapp.com/api/v1/quote

On a final note, i'd prefer not to get back the credentials/private key throughout the response...

[jperasmus]

I'll have to take a look at the warning you're receiving about the imports, it seems odd. My first thought is that there might be somewhere else that the full flamelink SDK is being imported, but we can look at that last, nothing too serious.

For the credentials being logged. In our Firestore implementation, we use Firestore document references for relational data, for example, each user object's permissions field is a reference to the actual document in the fl_permissions collection. This reference field includes all sorts of info, including the credentials. This happens only server-side because it runs in a trusted environment. To replace these references, you can use the populate option to replace these document references with the actual document object stored at that reference.

The easiest way to populate all and get rid of any references is to use populate: true. This is easier, but performance-wise, it is better to specify the exact fields you want to populate to avoid unnecessary processing being done.

The second part of your problem to pluck nested fields is something this SDK cannot currently do. We actually still have a TODO in the code for it. The idea would be that it works similar to the populate syntax where instead of just specifying an array of strings of the fields you want to pluck, you can pass an array of object with each it's own mandatory field property and optional nested fields again. This is something I would like to implement soon, but I cannot give you an exact date when I would be able to get to it because I am currently working on other features with higher priority.

For now, I would suggest populating the fields you are going to use and then write a quick field pluck function for your specific data to only pluck out the fields you want to return in your API.

[shaunnez]

Thanks for the information.

For now i'll write my own method to deep-pluck the fields out of the api response and populate fields accordingly. Hopefully if/when I get closer to production and performance optimization this will be something flamelink will support.

Cheers

[MLoth]

The second part of your problem to pluck nested fields is something this SDK cannot currently do. We actually still have a TODO in the code for it. The idea would be that it works similar to the populate syntax where instead of just specifying an array of strings of the fields you want to pluck, you can pass an array of object with each it's own mandatory field property and optional nested fields again. This is something I would like to implement soon, but I cannot give you an exact date when I would be able to get to it because I am currently working on other features with higher priority.

For now, I would suggest populating the fields you are going to use and then write a quick field pluck function for your specific data to only pluck out the fields you want to return in your API.

Any updates on this?