Import parts csv results in PicklingError: Can't pickle

[Ruudjhuu]

Please verify that this bug has NOT been raised before.

• [X] I checked and didn't find similar issue

Describe the bug*

When exporting a parts csv and then importing it with changes or new content a error pops up with: PicklingError: Can't pickle <function QuerySet.distinct at 0x7fd500b17c10>: it's not the same object as django.db.models.query.QuerySet.distinct

Steps to Reproduce

1. Go to: InvenTree Admin -> Home > Part > Parts > Export
2. Choose format csv and click SUBMIT
3. Go to: InvenTree Admin -> Home > Part > Parts > Import
4. Click Browse and chose the created file at step 2
5. Choose format csv
6. Click SUBMIT

Expected behavior

Data is imported and new/changed parts are available in the gui.

Deployment Method

• [X] Docker
• [ ] Bare metal

Version Information

Version Information:

InvenTree-Version: 0.8.4 Django Version: 3.2.15 Commit Hash: ca1fbf9 Commit Date: 2022-10-20 Database: postgresql Debug-Mode: False Deployed using Docker: True

Relevant log output

Line number: 1 - Can't pickle <function QuerySet.distinct at 0x7fd4fffba9d0>: it's not the same object as django.db.models.query.QuerySet.distinct
3, 2, , Transistors, , 1, 19.00000, 0, 0, 0, 0, 28, 2222, 0, 2222, , , , , part_images/trans_0q8jLZE.png, 0, 0, , 0, 1, 0, 1, 0, 1, 0, , 2022-11-21, 1, , 0.000, 1

Traceback (most recent call last):
File "/root/.local/lib/python3.9/site-packages/import_export/resources.py", line 668, in import_row
self.save_instance(instance, using_transactions, dry_run)
File "/root/.local/lib/python3.9/site-packages/import_export/resources.py", line 446, in save_instance
instance.save()
File "/home/inventree/InvenTree/part/models.py", line 419, in save
super().save(*args, **kwargs)
File "/root/.local/lib/python3.9/site-packages/mptt/models.py", line 1014, in save
super(MPTTModel, self).save(*args, **kwargs)
File "/root/.local/lib/python3.9/site-packages/django/db/models/base.py", line 739, in save
self.save_base(using=using, force_insert=force_insert,
File "/root/.local/lib/python3.9/site-packages/django/db/models/base.py", line 787, in save_base
post_save.send(
File "/root/.local/lib/python3.9/site-packages/django/dispatch/dispatcher.py", line 180, in send
return [
File "/root/.local/lib/python3.9/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
(receiver, receiver(signal=self, sender=sender, **named))
File "/home/inventree/InvenTree/part/models.py", line 2123, in after_save_part
InvenTree.tasks.offload_task(part_tasks.notify_low_stock_if_required, instance)
File "/home/inventree/InvenTree/InvenTree/tasks.py", line 86, in offload_task
task.run()
File "/root/.local/lib/python3.9/site-packages/django_q/tasks.py", line 719, in run
self.id = async_task(self.func, *self.args, **self.kwargs)
File "/root/.local/lib/python3.9/site-packages/django_q/tasks.py", line 69, in async_task
pack = SignedPackage.dumps(task)
File "/root/.local/lib/python3.9/site-packages/django_q/signing.py", line 15, in dumps
return signing.dumps(
File "/root/.local/lib/python3.9/site-packages/django/core/signing.py", line 110, in dumps
return TimestampSigner(key, salt=salt).sign_object(obj, serializer=serializer, compress=compress)
File "/root/.local/lib/python3.9/site-packages/django/core/signing.py", line 172, in sign_object
data = serializer().dumps(obj)
File "/root/.local/lib/python3.9/site-packages/django_q/signing.py", line 35, in dumps
return pickle.dumps(obj, protocol=pickle.HIGHEST_PROTOCOL)
_pickle.PicklingError: Can't pickle <function QuerySet.distinct at 0x7fd4fffba9d0>: it's not the same object as django.db.models.query.QuerySet.distinct

[SchrodingersGat]

@Ruudjhuu can you please provide a minimum working example of a .csv file which causes this issue?

[Ruudjhuu]

Of course, here are the files you need to be able to reproduce. I only substituted sensitive information.

Minimal input CSV:

import.csv

Docker compose:

Probably need to change the path to the nginx configuration. docker-compose.yaml.txt

env:

stack.env.txt

Nginx proxy conf:

nginx_conf

[bloemp]

Hi today I ran into the same problem, but I think I have some additional information that could help find/fix this... More file formats have this problem, it is not just CSV. I have also tested XLS and XLSX...

I wanted to explore this in my development environment, therefore I needed to dump my own "production" DB which is postgresql on docker (just like that of the OP) and import it into sqlite. The problem disapeared... Both "production" and development versions are on release 0.10 dev. Therefore, it seems that the problem is DB specific.

Additionally, it seems to be specific to some DB rows. When I export all parts from the postgresql file and immediately try to import it... it fails on a few rows... but not all...

Hope this helps to fix this problem (as the Part import wizard also seems broken - will make a separate issue for that)

[SchrodingersGat]

@bloemp interesting - can you share any more detailed information on the particular database rows that seem to be failing? Maybe there is something particular about those rows...

[bloemp]

@SchrodingersGat as far as i can tell right now, the rows themself (working/non working) don't seem to be particular different. My knowledge lacks at this point but as the stacktrace mentions the distinct funtion, and the django documentation explicitly mentions postgresql for some sort of behaviour... this could be somehow related. Never the less I will try to make a small test case downsizing my postgresql db to a minimum so you can investigate... but it will take me a few days.

[bloemp]

@SchrodingersGat well faster than expected... I have produced 2 files. One is the result of a datadump command from InvenTree running with the postgresql db that is completely stripped of data except for 2 parts and the part categories that they belong to. The second file is a CSV export of the parts from the admin page. If you try to import the CSV back again with a postgresql db you will get the exception... Loading the same datadump.json file into an InvenTree instance with a sqlite db (development env). Then try to import the CSV file, it works flawlessly.

Files Part import test case.zip

[SchrodingersGat]

Thanks @bloemp I think that this file helped me to find the bug

[motusRP]

I'm finding this bug in Inventree 0.15.3. Same error message:

Can't pickle <function QuerySet.distinct at 0x7fb2bc4762a0>: it's not the same object as django.db.models.query.QuerySet.distinct

I exported a CSV of all parts, then trimmed back to just the first part, and tried re-importing the same file, and this error pops up. If I export a block, edit, and re-import, I get one error per line in the import.

Other version info:

Version Information:

InvenTree-Version: 0.15.3 Django Version: 4.2.12 Commit Hash: d4d9aa9 Commit Date: 2024-05-27

Database: postgresql Debug-Mode: False Deployed using Docker: True Platform: Linux-5.15.0-107-generic-x86_64-with Installer: DOC

Active plugins: [{'name': 'InvenTreeBarcode', 'slug': 'inventreebarcode', 'version': '2.0.0'}, {'name': 'InvenTreeCoreNotificationsPlugin', 'slug': 'inventreecorenotificationsplugin', 'version': '1.0.0'}, {'name': 'InvenTreeCurrencyExchange', 'slug': 'inventreecurrencyexchange', 'version': '1.0.0'}, {'name': 'InvenTreeLabel', 'slug': 'inventreelabel', 'version': '1.0.0'}, {'name': 'InvenTreeLabelMachine', 'slug': 'inventreelabelmachine', 'version': '1.0.0'}, {'name': 'InvenTreeLabelSheet', 'slug': 'inventreelabelsheet', 'version': '1.0.0'}, {'name': 'DigiKeyPlugin', 'slug': 'digikeyplugin', 'version': '1.0.0'}, {'name': 'LCSCPlugin', 'slug': 'lcscplugin', 'version': '1.0.0'}, {'name': 'MouserPlugin', 'slug': 'mouserplugin', 'version': '1.0.0'}, {'name': 'TMEPlugin', 'slug': 'tmeplugin', 'version': '1.0.0'}]

[motusRP]

Update: I reviewed all of the part data and I can't see any issues with the re-imported part, the changes I make in the CSV before re-importing seem to be incorporated (tried changing IPN and Revision) and there doesn't seem to be any bad data anywhere. So maybe this is just an erroneous error message?

[SchrodingersGat]

@motusRP can you provide a full stack trace for the error?

You should be able to see this error in the admin interface - https://docs.inventree.org/en/stable/settings/logs/