Importing products from magento gives empty results if first product ID is greater than 600 (eg 686)

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Import 650 products in Magento. 
2. Delete the 650 products from magento's management
3. Import any more products in Magento so that the product ID starts at a 
number greater than 600.
4. Try to import the products from Magento to OpenERP

What is the expected output? What do you see instead?

Normally the module should be able to detect the products and import them. 
However, it starts to search from Product ID 1 to 300 and up to ID 600 and it 
finds no products. Then it halts because the result set is empty.

In the log file the result is like this.

Logged in to Magento
Products ID from 1 to 300
Last import: "timestamp"
Done. 

However if you add 1 or more products with IDs within the range 1-300 the 
result log is  like this:

Logged in to Magento
Products ID from 1 to 300
Last import: "timestamp"
Loading product: ......
etc etc
Products ID from 301 to 600 
Last import: "timestamp"
Done.

What version of the product are you using? On what operating system?

I am using Magento 1.7 and OpenERP 6.1. Tested on windows 7 32bit/64bit and 
Windows 8 64bit. Webserver is Linux based. 

Please provide any additional information below.

Based on my observations it seems that the module is creating batches of 300 
product IDs and if it doesn't find any products within the range it halts. If 
it finds at least one record it creates a second batch of the next 300 product 
IDs. 
In my opinion in shops with a lot of products it's entirely possible that a 
whole series of old products ranging from Product ID 1 to 300 might get deleted 
and then the synchronization will start failing.

I would suggest if it is at all possible to first check for the minimum 
existing product ID and the maximum existing product ID in the magento store. 
After that it should create batches of 300 Product IDs until the maximum 
product ID is reached and then stop execution. 

Thanks for your help in advance.

Original issue reported on code.google.com by gab...@gmail.com on 11 Sep 2012 at 8:04

[GoogleCodeExporter]

Also I forgot to mention I'm using the versions of sneldev-magento-openerp I 
downloaded from here

sneldev_magento_openerp-0.9.2.6-openerp.zip
sneldev_magneto_openerp-0.7.5-magento.zip 

Original comment by gab...@gmail.com on 11 Sep 2012 at 12:47

[GoogleCodeExporter]

It seems I am having the same problem.  When we first setup Magento, we 
imported about 1,300 products but the CSV file had some error.  So we deleted 
all products, updated the CSV file, and import again.  Now our lowest product 
ID is somewhere around 1,350... and the connector will behave exactly like you 
mentioned:  try to import first 300 products, and stop.  My log entries are:

2012-10-22 07:37:52 ===== Products import
2012-10-22 07:37:53 Logged in to Magento
2012-10-22 07:37:53 Products ID from 1 to 300
2012-10-22 07:37:53 Last import: 2012-10-22 07:22:41.12

Now, on the sneldev_magento.py file (path 
/opt/openerp/server/openerp/addons/sneldev_magento/ on my installation), line 
574, which is part of the 'Product import' routine, it says "increment = 300".  
I changed it to 3000 just for kicks... and when I run the import, it takes 
longer and the log entry now says:

2012-10-22 07:42:08 ===== Products import
2012-10-22 07:42:08 Logged in to Magento
2012-10-22 07:42:08 Products ID from 1 to 3000
2012-10-22 07:42:11 Last import: 2012-10-22 07:40:09.46
2012-10-22 07:42:11 Products ID from 3001 to 6000
2012-10-22 07:42:12 Last import: 2012-10-22 07:40:09.46
2012-10-22 07:42:12 Done

BUT - I still have no products in OpenERP.  The import process obviously 
behaves differently now... after the first 3000 it did try the next 3000 (from 
3001 to 6001), whereas previously it would simply stop at 300.  So it found my 
products.  At that point I thought maybe there are some other places that also 
limits 300 products per batch (say, declaration of variable/array).  So I went 
to the .py file and changed "increment" back to 300, but "index" to 1100.  This 
in effect force the import to "start" from ProductID 1100.  My new log entry 
looks like this:

2012-10-22 07:55:07 ===== Products import
2012-10-22 07:55:07 Logged in to Magento
2012-10-22 07:55:07 Products ID from 1100 to 1399
2012-10-22 07:55:08 Last import: 2012-10-22 07:54:03.99
2012-10-22 07:55:08 Products ID from 1400 to 1699
2012-10-22 07:55:09 Last import: 2012-10-22 07:54:03.99
2012-10-22 07:55:09 Products ID from 1700 to 1999
2012-10-22 07:55:11 Last import: 2012-10-22 07:54:03.99
2012-10-22 07:55:11 Products ID from 2000 to 2299
2012-10-22 07:55:12 Last import: 2012-10-22 07:54:03.99
2012-10-22 07:55:12 Products ID from 2300 to 2599
2012-10-22 07:55:13 Last import: 2012-10-22 07:54:03.99
2012-10-22 07:55:13 Products ID from 2600 to 2899
2012-10-22 07:55:13 Last import: 2012-10-22 07:54:03.99
2012-10-22 07:55:13 Done

So, looking good and the theory seems to work... but I STILL have no products 
in OpenERP.

Since all the categories imported fine, I have to think the actual connection 
is working... but this is getting the better of me.  However, I might be 
running into a different problem.  Can you try changing your .py file and 
increase your "increment" to a higher number, and see if you are getting the 
products to import?  Also remember after changing the .py file you need to 
restart OpenERP (sudo /etc/init.d/openerp-server restart) or the change will 
not take effect.

Cheers,

Wallace

Original comment by Wallace...@gmail.com on 22 Oct 2012 at 8:03

[GoogleCodeExporter]

By the way I am also using Magento 1.7.02 and OpenERP 6.1.  Both servers are 
Ubuntu 12.04 LTS.  Tried import from both web client and Windows client.

Thanks,

Original comment by Wallace...@gmail.com on 22 Oct 2012 at 8:06

[GoogleCodeExporter]

I've already done that and it didn't work for me either so I gave up and 
truncated my magento database since it was a fresh magento installation. Then 
it proceeded normally. However the problem was exactly as you described after 
you change the increment in python. The actual number of how many product ids 
it's checking is probably defined elsewhere. 

If you're starting the store from scratch like me and it doesn't contain actual 
data from a previous installation I recommend you do the same as me following 
this guide and truncate all product related data. Then the new correct import 
will start from ID 1. 

http://www.magentocommerce.com/boards/viewthread/20553/

After that the connector will work for you again but this is just to get you 
started. The real problem is not solved and is beyond my current knowledge of 
python or magento api. Although this connector appears to not be worked on 
anymore I was hoping someone else might have an idea how to fix it so I posted 
here just in case. 

Original comment by gab...@gmail.com on 22 Oct 2012 at 9:16

[GoogleCodeExporter]

Found out why it didn't work... when we first tried the import (and failed 
because the script only tried 300 products), it wrote a timestamp into the 
OpenERP DB.  After we modified the script, and re-attempt import, the script 
found all the new products but their timestamp are still *older* than what was 
written in the DB... thinking it will create duplicated entries, the script 
skipped all products.

If I create a new DB in OpenERP, change the script BEFORE attempting the first 
import, it will work.  I now have all our products in OpenERP even though the 
first product ID is #1175.

Hope this will be of help to someone else.

Cheers,

Wallace

Original comment by zshot....@gmail.com on 29 Oct 2012 at 12:24

[GoogleCodeExporter]

So tell me if I understood correctly. 
You changed the increment in python to let's say 1500 and then restarted 
openerp server. After that you setup a new DB for openerp and then retried the 
import and it worked because there were no timestamps from previous failed 
operations? 

I'm glad you found a better workaround. :)

Original comment by gab...@gmail.com on 29 Oct 2012 at 12:33

[GoogleCodeExporter]

Yup that pretty much sums it up.  If the script is modified and ran while the 
DB is still "virgin", it works and will continue to work thereafter.  =)

W.

Original comment by zshot....@gmail.com on 29 Oct 2012 at 11:55