Multi process/Multi request's problem

[Tintin4257]

Hi,

Since several months, we have been using Djongo as a database engine for our APIs in Django RestFramework. Sometimes we noticed irregularities in answers of our APIs: data are different from one request to another, bad authentication for no apparent reason, etc.... (especially when many request are done at the same time) I looked deeper at the problem, and it seems that the overlay applied by Djongo (i.e. transforming sql request into pymongo request and sending the result back to the model manager) creates a problem of alignment for the parameters in different requests. Request run simultaneously and in a large numbers of them.

views.py

Here are the 2 views I call ...

class CurrentMyUserView(generics.RetrieveUpdateAPIView):
    serializer_class = MyUserSerializer
    permission_classes = (permissions.IsAuthenticated,)

    def get_object(self):
        try:
            queryset = MyUser.objects.get(user=self.request.user)
        except(MyUser.DoesNotExist):
            raise exceptions.APIException()
        return queryset

class ThemeViewSet(viewsets.ModelViewSet):
    queryset = Theme.objects.all()
    serializer_class = ThemeSerializer
    permission_classes = (permissions.IsAuthenticatedOrReadOnly,)

call_request.py

... using this program

from multiprocessing import Pool, Process
from scripts_config import *

def call_request(i):
    max_len = 10
    f = open("log_user{0}.txt".format(i), "w+")
    for i in range(0, max_len):
        r = requests.get(API_URL + 'users/myuser/', auth=ApiAuth(token))
        if r.status_code != 200:
            f.write("{0} {1}".format(str(r.status_code), r.content.decode()))
        else:
            f.write("{0} {1}".format(str(json.loads(r.content)['username']), json.loads(r.content)['email']))
        r = requests.get(API_URL + 'resources/themes/', auth=ApiAuth(token))
        f.write("\n")
    f.close()

if __name__ == '__main__':
    with Pool(5) as p:
        p.map(call_request, range(0, 5))

Nothing special so. But the result is strange: while I should get 10 times my user name (admin here), sometimes I crash (500), sometimes I am not authenticated (401), and sometimes I even get the name of another user.

Example: log1.txt

admin 
i2rb5g+9b0y58wai5jn0@sharklasers.com i2rb5g+9b0y58wai5jn0@sharklasers.com
admin 
admin 
admin 
admin 
401 {"detail":"Invalid token."}
admin 
admin 
admin 

The same test with a "classic" database engine from django gives a perfect result (only code 200 and only "admin" in file)

So I put some print() in djongo to see what's wrong:

converters.py

#line 137
class WhereConverter(Converter):
    nested_op: 'WhereOp' = None
    op: 'WhereOp' = None

    def parse(self):
        sm = self.query.statement
        tok = sm[self.begin_id]
        print("HERE MY PARAMS: {0}".format(self.query.params))
        self.op = WhereOp(
            token_id=0,
            token=tok,
            query=self.query,
            params=self.query.params
        )
        self.end_id = self.begin_id

    def to_mongo(self):
        return {'filter': self.op.to_mongo()}

query.py

#line 95
class SelectQuery(Query):
[...]
    def parse(self):
        tok_id = 0
        tok = self.statement[0]

        print("THIS IS MY STATEMENT: {0}".format(self.statement.__str__()))
        while tok_id is not None:
[...]
    def __iter__(self):
[...]
        for doc in cursor:
            print(doc)
            yield self._align_results(doc)
        return

By redirecting these prints into a file I then understood the origin of the problem:

output.txt

[...]
THIS IS MY STATEMENT: SELECT "auth_user"."id", "auth_user"."password", "auth_user"."last_login", "auth_user"."is_superuser", "auth_user"."username", "auth_user"."first_name", "auth_user"."last_name", "auth_user"."email", "auth_user"."is_staff", "auth_user"."is_active", "auth_user"."date_joined" FROM "auth_user" WHERE "auth_user"."id" = %(0)s
HERE MY PARAMS: (1,)
THIS IS MY STATEMENT: SELECT "Resources_subcategory"."id", "Resources_subcategory"."is_available", "Resources_subcategory"."subcategory_name_en", "Resources_subcategory"."subcategory_name_en_us", "Resources_subcategory"."subcategory_name_fr", "Resources_subcategory"."subcategory_nb_articles", "Resources_subcategory"."subcategory_category_id" FROM "Resources_subcategory" WHERE "Resources_subcategory"."subcategory_category_id" = %(0)s
HERE MY PARAMS: (22,)
THIS IS MY STATEMENT: SELECT "Resources_category"."id", "Resources_category"."is_available", "Resources_category"."category_name_en", "Resources_category"."category_name_en_us", "Resources_category"."category_name_fr", "Resources_category"."category_nb_articles", "Resources_category"."category_theme_id" FROM "Resources_category" WHERE "Resources_category"."category_theme_id" = %(0)s
HERE MY PARAMS: (21,)
OrderedDict([('_id', ObjectId('5d95b8a7a488ea15f52654c4')), ('id', 73), ('is_available', True), ('category_name_en', 'Miscellaneous'), ('category_name_en_us', 'Miscellaneous'), ('category_name_fr', 'Divers'), ('category_nb_articles', 0), ('category_theme_id', 30)])
THIS IS MY STATEMENT: SELECT "Resources_subcategory"."id", "Resources_subcategory"."is_available", "Resources_subcategory"."subcategory_name_en", "Resources_subcategory"."subcategory_name_en_us", "Resources_subcategory"."subcategory_name_fr", "Resources_subcategory"."subcategory_nb_articles", "Resources_subcategory"."subcategory_category_id" FROM "Resources_subcategory" WHERE "Resources_subcategory"."subcategory_category_id" = %(0)s
HERE MY PARAMS: (73,)
OrderedDict([('_id', ObjectId('5d95b89ba488ea15f52653dc')), ('id', 20), ('is_available', True), ('subcategory_name_en', 'Small Appliances'), ('subcategory_name_en_us', 'Small Appliances'), ('subcategory_name_fr', 'Petit Électroménager'), ('subcategory_nb_articles', 0), ('subcategory_category_id', 22)])
THIS IS MY STATEMENT: SELECT "Resources_article"."id", "Resources_article"."is_available", "Resources_article"."article_name", "Resources_article"."article_ref", "Resources_article"."article_score", "Resources_article"."article_photo1", "Resources_article"."article_photo2", "Resources_article"."article_photo3", "Resources_article"."article_photo4", "Resources_article"."article_photo5", "Resources_article"."article_manufacturer", "Resources_article"."article_manufacturer_ref", "Resources_article"."article_brand", "Resources_article"."article_model_number", "Resources_article"."article_public", "Resources_article"."article_id_product", "Resources_article"."article_description", "Resources_article"."article_characteristics", "Resources_article"."article_is_box", "Resources_article"."article_subcategory_id" FROM "Resources_article" WHERE "Resources_article"."article_subcategory_id" = %(0)s
HERE MY PARAMS: (20,)
OrderedDict([('_id', ObjectId('5d9cbb8a00093231c5dd86a0')), ('id', 22), ('password', '****'), ('last_login', datetime.datetime(2019, 10, 14, 18, 23, 44, 723000)), ('is_superuser', False), ('username', 'i2rb5g+9b0y58wai5jn0@sharklasers.com'), ('first_name', 'Raoul'), ('last_name', 'Laser'), ('email', 'i2rb5g+9b0y58wai5jn0@sharklasers.com'), ('is_staff', False), ('is_active', True), ('date_joined', datetime.datetime(2019, 10, 8, 16, 38, 34, 42000))])
[...]

It seems that the User object I get (id=22, alias i2rb5g+9b0y58wai5jn0@sharklasers.com) is not the one I request when I make my SQL request (id=1, alias admin) because of the interference of requests made in parallel for my themes.

Thx for reading and sorry for the long post ! I hope it help

[PDK2020]

Hello Tintin, I think this issue is related to this issue also:

https://github.com/nesdis/djongo/issues/348

Have you reached to a solutions? I think that the problem is that Djongo is no managing well large number of threads.

[Tintin4257]

Hi PDK,

For the moment we are still managing the crisis with a lot of try except in the views (a band-aid in short).

But in my spare time I'm working on a more definitive solution.

[PDK2020]

Hi guys, We have been debugin the version 1.2.33 and we have developed a fix for this error. We have tested this fix in two ways: a) Streessing the application in testing with some scripts that simulate a very high demand scenario b) In Production environment.

The fix works perfectly. Can you use it in the new versions so it can be used widely in the community?

We have edited two files query.py and operators.py. Here are the two files: https://drive.google.com/drive/folders/1VORvPO9GKtR2K8BkeAlsMoVQNHuBnWiN?usp=sharing

Thanks and hope it will be useful!

Bioengineer Patricio Donnelly Kehoe, MSc. PhD. CTO at Kozaca SA, Argentina

[Raznak]

Hi guys, We have been debugin the version 1.2.33 and we have developed a fix for this error. We have tested this fix in two ways: a) Streessing the application in testing with some scripts that simulate a very high demand scenario b) In Production environment.

The fix works perfectly. Can you use it in the new versions so it can be used widely in the community?

We have edited two files query.py and operators.py. Here are the two files: https://drive.google.com/drive/folders/1VORvPO9GKtR2K8BkeAlsMoVQNHuBnWiN?usp=sharing

Thanks and hope it will be useful!

Bioengineer Patricio Donnelly Kehoe, MSc. PhD. CTO at Kozaca SA, Argentina

Can you publish a Pull request with the fix ?

[albadrun]

Can someone publish this fix please :') Need to use @PDK2020 patch.

[PDK2020]

I have tried to make a pull request but I cannot find the version 1.2.33 in the repository. I think Master is too different to the version we have worked on. I have also checked with version 1.2.23 but it is also very different.