Cross-referencing models not supported

Alain1405 commented 8 years ago

Hi (it's me), The client works properly if I query for a resource, let's say A, that has a ToMany relationship with B, if B has no references to A. However, if this condition is not verified, the browser freezes and I can see in the Network Chrome dev tab that while all request to A resources are performed properly (return 200), for B resources the requests are marked as pending (OPTION requests pass, but GET are marked pending).

Backend:

class A(ModelResource):
    class Meta:
        model = A
        include_type = True
        include_id = True

     class Schema:
         b = fields.ToMany('b')

class B(ModelResource):
    class Meta:
        model = B
        include_type = True
        include_id = True

     class Schema:
         a = fields.ToMany('a')

and in Angular:

...
.factory('A', ['potion', function (potion)
    {
        function A()
        {
            Item.call(this);
        }

        A.prototype = Object.create(Item.prototype);
        A.query = Item.query;
        A.fetch = Item.fetch;

        potion.register('/a', A);

        return A;
    }])

.factory('B', ['potion', function (potion)
    {

        function B()
        {
            Item.call(this);
        }

        B.prototype = Object.create(Item.prototype);
        B.query = Item.query;
        B.fetch = Item.fetch;

        potion.register('/b', B);

        return B;
    }])

... controller

A.query().then(function (a)
   {
        console.log('A');
        console.log(a);
    });

The promise is never resolved ('A') is never printed.

rolandjitsu commented 8 years ago

@Alain1405 ok. I'm not sure if this is a client issue though. Do you mean that the get never get's resolved and it's still pending?

Or do you actually get an error?

Alain1405 commented 8 years ago

On the backend log I can see that all requests are successful. Chrome however freezes and requests to B are marked as pending, as if the response wasn't received.

Here's my 2 cents: What we see is only the first requests to resources that are not cached by potion-node. Potion-node resolves all A resources, then it passes to the references. It queries all B resources referenced by A and finds references to A. This causes an infinite loop that I don't know how to debug.

rolandjitsu commented 8 years ago

@Alain1405 I'll write a test for this scenario and come back to you, on Monday probably.

Alain1405 commented 8 years ago

Cool, thanks. Have a nice weekend

Alain1405 commented 8 years ago

Also, I have tried to add a method to pull the schema:

function _createResource(key, name) {
    var Resource = new Function(
        'Item',
        'return function ' + name + '(properties) {Item.call(this, properties);}'
    )(Item);

    Resource.prototype = Object.create(Item.prototype);
    Resource.query = Item.query;
    Resource.fetch = Item.fetch;
    Resource.first = Item.first;

    Resource.getSchema = Route.GET('/schema');

    potion.register('/' + key, Resource);
}

It logs the error Uninterpretable or unknown resource URI: # because Potion schemas have a self reference property

"links":{
    "targetSchema": {
            "$ref": "#"
          }
...
}

rolandjitsu commented 8 years ago

@Alain1405 could you open a new issue for the self reference problem?

lyschoening commented 8 years ago

@Alain1405 I can see you are working on auto-generating the potion models from the schema. To resolve issues like this one and others you might come across it would be a good idea for you to look at the Python version of the client at https://github.com/biosustain/potion-client.

The Python client addresses the issue you are having here by resolving all schema refs with a referrer: https://github.com/biosustain/potion-client/blob/master/potion_client/converter.py#L115:L155

Alain1405 commented 8 years ago

@rolandjitsu this only fixed partially the issue. The browser doesn't indeed freeze anymore but not all promises are resolved it seems. The following never logs to console.

A.query().then(function(resp){
console.log("You wish");
})

I've spent quite some time debugging it and the issues sees to be as follow: Resource A1 has multiple references to B resources, something like a1={Bs: [{$ref: b1}, {$ref: b2}, {$ref: b3}]}, Also B resources reference multiple A resources: b1={$ref: a1}, {$ref: a2}, {$ref: a3}.

First I query for all As.
Once the response is back ( [a1, a2, a3]), potion-node passes a1 to _fromPotionJSON.
_fromPotionJSON requests all references to Bs, 1 by 1.
Once the first request to b1 returns, b1 is passed to _fromPotionJSON, which queries for As references (yes, those could be cached in the first query).

At this stage, b1 is still marked as pending and it won't resolve until all inner references are resolved, but this won't happen because inner references include a1 which references b1 which is pending.

Not sure what the best solution would be. You should possibly resolve 1 level of $refs only.

Alain1405 commented 8 years ago

@lyschoening I don't quite get it. I need the data on the frontend. The python client let me query the potion api in a simple way, but I will need the frontend to query a the python client interface that would then query the potion API. o.O Is this what you mean or there is some piece missing?

rolandjitsu commented 8 years ago

@Alain1405 I just pushed a change on npm. Could you give it a try and confirm if it works now?

I've also added a test suite for testing Item.query() where the resolved data contains resources with cross references.

Alain1405 commented 8 years ago

Still not working. Here you have some logging, hoping that it will be useful to you. At the bottom the diff files show what is being logged.:

// Querying A
angular.js:33 Angular fetched http:/127.0.0.1:8888/api/A
base.js:301 Fetched, stage 1 /api/A
// Resolving a1 $refs
base.js:353 Added to pending /api/B/1
base.js:353 Added to pending /api/B/2
base.js:353 Added to pending /api/B/3
base.js:353 Added to pending /api/B/4
base.js:353 Added to pending /api/B/5
base.js:353 Added to pending /api/B/6
base.js:353 Added to pending /api/B/7
base.js:353 Added to pending /api/B/8
base.js:353 Added to pending /api/B/9
base.js:353 Added to pending /api/B/10
// Resolving a2 $refs (and so on so on)
base.js:336 Found pending /api/B/1
base.js:336 Found pending /api/B/3
base.js:336 Found pending /api/B/4
base.js:336 Found pending /api/B/5
base.js:336 Found pending /api/B/2
base.js:336 Found pending /api/B/6
base.js:336 Found pending /api/B/7
base.js:336 Found pending /api/B/8
base.js:336 Found pending /api/B/9
base.js:336 Found pending /api/B/10

// Resolving a3 $refs
base.js:336 Found pending /api/B/1
base.js:336 Found pending /api/B/2
base.js:336 Found pending /api/B/3
base.js:336 Found pending /api/B/4
base.js:336 Found pending /api/B/5
base.js:336 Found pending /api/B/6
base.js:336 Found pending /api/B/7
base.js:336 Found pending /api/B/8
base.js:336 Found pending /api/B/9
base.js:336 Found pending /api/B/10

// Resolving a4 $refs
base.js:336 Found pending /api/B/2
base.js:336 Found pending /api/B/1
base.js:336 Found pending /api/B/4
base.js:336 Found pending /api/B/5
base.js:336 Found pending /api/B/3
base.js:336 Found pending /api/B/6
base.js:336 Found pending /api/B/7
base.js:336 Found pending /api/B/8
base.js:336 Found pending /api/B/9
base.js:336 Found pending /api/B/10

// First request for B ref returns
angular.js:33 Angular fetched http:/127.0.0.1:8888/api/B/2
base.js:301 Fetched, stage 1 /api/B/2

// Resolving b2 $refs
base.js:353 Added to pending /api/A/4
base.js:353 Added to pending /api/A/20
base.js:353 Added to pending /api/A/7
base.js:353 Added to pending /api/A/9
base.js:353 Added to pending /api/A/13
base.js:353 Added to pending /api/A/6

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/B/6
base.js:301 Fetched, stage 1 /api/B/6

// Resolving b2 $refs (and so on so on)
base.js:336 Found pending /api/A/20
base.js:336 Found pending /api/A/4
base.js:336 Found pending /api/A/13
base.js:336 Found pending /api/A/7
base.js:336 Found pending /api/A/9
base.js:336 Found pending /api/A/6
base.js:353 Added to pending /api/A/10

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/B/4
base.js:301 Fetched, stage 1 /api/B/4
base.js:336 Found pending /api/A/20
base.js:336 Found pending /api/A/4
base.js:336 Found pending /api/A/7
base.js:336 Found pending /api/A/9
base.js:336 Found pending /api/A/13
base.js:336 Found pending /api/A/6
base.js:336 Found pending /api/A/10

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/B/3
base.js:301 Fetched, stage 1 /api/B/3
base.js:336 Found pending /api/A/20
base.js:336 Found pending /api/A/4
base.js:336 Found pending /api/A/13
base.js:336 Found pending /api/A/9
base.js:336 Found pending /api/A/6
base.js:336 Found pending /api/A/7
base.js:336 Found pending /api/A/10

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/B/1
base.js:301 Fetched, stage 1 /api/B/1
base.js:336 Found pending /api/A/4
base.js:336 Found pending /api/A/20
base.js:336 Found pending /api/A/13
base.js:336 Found pending /api/A/7
base.js:336 Found pending /api/A/9
base.js:336 Found pending /api/A/6

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/B/5
base.js:301 Fetched, stage 1 /api/B/5
base.js:336 Found pending /api/A/20
base.js:336 Found pending /api/A/9
base.js:336 Found pending /api/A/7
base.js:336 Found pending /api/A/13
base.js:336 Found pending /api/A/4
base.js:336 Found pending /api/A/6

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/B/7
base.js:301 Fetched, stage 1 /api/B/7
base.js:336 Found pending /api/A/20
base.js:336 Found pending /api/A/4
base.js:336 Found pending /api/A/13
base.js:336 Found pending /api/A/7
base.js:336 Found pending /api/A/9
base.js:336 Found pending /api/A/10

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/B/8
base.js:301 Fetched, stage 1 /api/B/8
base.js:336 Found pending /api/A/20
base.js:336 Found pending /api/A/4
base.js:336 Found pending /api/A/13
base.js:336 Found pending /api/A/7
base.js:336 Found pending /api/A/9
base.js:336 Found pending /api/A/10

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/B/9
base.js:301 Fetched, stage 1 /api/B/9
base.js:336 Found pending /api/A/20
base.js:336 Found pending /api/A/4
base.js:336 Found pending /api/A/13
base.js:336 Found pending /api/A/7
base.js:336 Found pending /api/A/9
base.js:336 Found pending /api/A/10

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/B/10
base.js:301 Fetched, stage 1 /api/B/10
base.js:336 Found pending /api/A/20
base.js:336 Found pending /api/A/4
base.js:336 Found pending /api/A/13
base.js:336 Found pending /api/A/7
base.js:336 Found pending /api/A/9
base.js:336 Found pending /api/A/10

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/A/4
base.js:301 Fetched, stage 1 /api/A/4
base.js:336 Found pending /api/B/1
base.js:336 Found pending /api/B/2
base.js:336 Found pending /api/B/3
base.js:336 Found pending /api/B/4
base.js:336 Found pending /api/B/5
base.js:336 Found pending /api/B/6
base.js:336 Found pending /api/B/7
base.js:336 Found pending /api/B/8
base.js:336 Found pending /api/B/9
base.js:336 Found pending /api/B/10

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/A/20
base.js:301 Fetched, stage 1 /api/A/20
base.js:336 Found pending /api/B/1
base.js:336 Found pending /api/B/2
base.js:336 Found pending /api/B/3
base.js:336 Found pending /api/B/4
base.js:336 Found pending /api/B/5
base.js:336 Found pending /api/B/6
base.js:336 Found pending /api/B/7
base.js:336 Found pending /api/B/8
base.js:336 Found pending /api/B/9
base.js:336 Found pending /api/B/10

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/A/9
base.js:301 Fetched, stage 1 /api/A/9
base.js:336 Found pending /api/B/2
base.js:336 Found pending /api/B/1
base.js:336 Found pending /api/B/4
base.js:336 Found pending /api/B/5
base.js:336 Found pending /api/B/3
base.js:336 Found pending /api/B/6
base.js:336 Found pending /api/B/7
base.js:336 Found pending /api/B/8
base.js:336 Found pending /api/B/9
base.js:336 Found pending /api/B/10

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/A/7
base.js:301 Fetched, stage 1 /api/A/7
base.js:336 Found pending /api/B/1
base.js:336 Found pending /api/B/2
base.js:336 Found pending /api/B/4
base.js:336 Found pending /api/B/5
base.js:336 Found pending /api/B/3
base.js:336 Found pending /api/B/6
base.js:336 Found pending /api/B/7
base.js:336 Found pending /api/B/8
base.js:336 Found pending /api/B/9
base.js:336 Found pending /api/B/10

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/A/13
base.js:301 Fetched, stage 1 /api/A/13
base.js:336 Found pending /api/B/1
base.js:336 Found pending /api/B/3
base.js:336 Found pending /api/B/4
base.js:336 Found pending /api/B/5
base.js:336 Found pending /api/B/2
base.js:336 Found pending /api/B/6
base.js:336 Found pending /api/B/7
base.js:336 Found pending /api/B/8
base.js:336 Found pending /api/B/9
base.js:336 Found pending /api/B/10

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/A/6
base.js:301 Fetched, stage 1 /api/A/6
base.js:336 Found pending /api/B/4
base.js:336 Found pending /api/B/2
base.js:336 Found pending /api/B/1
base.js:336 Found pending /api/B/3
base.js:336 Found pending /api/B/5
base.js:336 Found pending /api/B/6

angular.js:33 Angular fetched http:/127.0.0.1:8888/api/A/10
base.js:301 Fetched, stage 1 /api/A/10
base.js:336 Found pending /api/B/6
base.js:336 Found pending /api/B/3
base.js:336 Found pending /api/B/4
base.js:336 Found pending /api/B/7
base.js:336 Found pending /api/B/8
base.js:336 Found pending /api/B/9
base.js:336 Found pending /api/B/10

base_diff.txt

angular_diff.txt

rolandjitsu commented 8 years ago

@Alain1405 as you can see that tests are passing for the new scenario I added. But maybe I misunderstood the scenario you were describing.

Also, adding some logs does not really help understand what you are experiencing. I will try to see if using the Angular 1 version would somehow behave differently.

lyschoening commented 8 years ago

I might have misunderstood what you wanted to do. I just wanted to point out if you want to generate your classes from the schema, all of this is done in the Python client so you can use it as inspiration when building your JavaScript implementation.

On Jun 28, 2016 12:14 AM, "Alain Scialoja" notifications@github.com wrote:

@lyschoening https://github.com/lyschoening I don't quite get it. I need the data on the frontend. The python client let me query the potion api in a simple way, but I will need the frontend to query a the python client interface that would then query the potion API. o.O Is this what you mean or there is some piece missing?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/biosustain/potion-node/issues/10#issuecomment-228892232, or mute the thread https://github.com/notifications/unsubscribe/AAEhZc0JE1OtfS1j3hELkkfhhbOLEnUqks5qQEs9gaJpZM4I-W4i .

Alain1405 commented 8 years ago

@rolandjitsu the logs show how the recursion gets stuck in the chain of promises. The diff files show where the logs are coming from. I had a look at the tests and it looks like the relationships are actually the same scenario as mine, but indeed the tests don't cover the angular part and the promises are resolved immediately, which should make a difference. @lyschoening I am the one who misunderstood. Thanks for the references, I will have a look at your code!

rolandjitsu commented 8 years ago

@Alain1405 I'll test the angular implementation today to see what is going on there. It would be helpful if you could setup a plunk with the setup you have.

rolandjitsu commented 8 years ago

@Alain1405 I just added another test suite for Angular 1. I still cannot see any issues.

Note that I am working in a ES6 env. Maybe that has something to do with it. As I said, sharing some code would definitely help me see what it is that is going wrong.

Alain1405 commented 8 years ago

@rolandjitsu I've tried to create a working requirebin with no success. Here is an extract of my code:

var angular = require('angular');
require('angular-resource');
require("reflect-metadata");
require("potion-client/angular");

var Item = require("potion-client/angular").Item;

var app = angular.module('app', []);

app.factory('crudClient', ['potion', '$resource', function (potion, $resource)
{
    var models_map = {};

    function _createResource(name)
    {
        var Resource = new Function(
            'Item',
            'return function ' + name + '(properties) {Item.call(this, properties);}'
        )(Item);

        Resource.prototype = Object.create(Item.prototype);
        Resource.query = Item.query;
        Resource.fetch = Item.fetch;
        Resource.first = Item.first;

        // Resource.getSchema = Route.GET('/schema');
        Resource.getSchema = function ()
        {
            return $resource('http://localhost:8888/api/' + name + '/schema').get().$promise;
        };

        potion.register('/' + name, Resource);
        models_map[name] = Resource;
    }

    _createResource('A');
    _createResource('B');

    return {
        getResource: function (res_name)
        {
            return models_map[res_name]
        }
    };
}]);

app.controller('MainController',
    [
        '$scope',
        'crudClient',
        function ($scope, crudClient)
        {

            var A = crudClient.getResource('A');

            A.query(
                {
                    page: 1,
                    perPage: 5
                },
                {
                    paginate: true
                })
                .then(function (data)
                    {
                        console.log(data._items);
                    },
                    function (err)
                    {
                        console.log('Error getting resource: ' + err);
                    });
        }]);

I also get the following error:

angular.js:13708 TypeError: Cannot read property 'save' of undefined
    at Item.save (base.js:116)
    at Scope.$scope.saveResource (resources-controller.js:96)

when doing

    var new = new A;
    new.save();

Alain1405 commented 8 years ago

Finally you were right. I changed the workflow to include ES6 scripts and everything works properly. Unless you want to extend this package to EC5, this issue can be close for what concerns me.

rolandjitsu commented 8 years ago

@Alain1405 good to hear that. I'll open an issue, but I cannot promise there will be support for ES5 soon.

biosustain / potion-node

Cross-referencing models not supported #10