bulk upload of a large product catalog: max_stack_depth too low

[GoogleCodeExporter]

while  I was trying to synchronize 20000 products, I got the following error. 

Thu, 18 Sep 2008 14:48:25 ERROR:web-services:Exception in call: Traceback
(most recent call last):
  File "/usr/lib/python2.4/site-packages/tinyerp-server/netsvc.py", line
200, in _dispatch
    r=m(*params)
  File
"/usr/lib/python2.4/site-packages/tinyerp-server/service/web_services.py",
line 422, in execute
    return self._execute(db, uid, wiz_id, datas, action, context)
  File
"/usr/lib/python2.4/site-packages/tinyerp-server/service/web_services.py",
line 402, in _execute
    return wiz.execute(db, uid, self.wiz_datas[wiz_id], action, context)
  File
"/usr/lib/python2.4/site-packages/tinyerp-server/wizard/__init__.py", line
160, in execute
    res = self.execute_cr(cr, uid, data, state, context)
  File
"/usr/lib/python2.4/site-packages/tinyerp-server/wizard/__init__.py", line
75, in execute_cr
    action_res = action(self, cr, uid, data, context)
  File
"/usr/lib/python2.4/site-packages/tinyerp-server/addons/magento_openerp_smile.zi
p/magento_openerp_smile/wizard/magento_product_synchronize.py",
line 87, in _do_export
  File "/usr/lib/python2.4/site-packages/tinyerp-server/osv/orm.py", line
196, in __getattr__
    return self[name]
  File "/usr/lib/python2.4/site-packages/tinyerp-server/osv/orm.py", line
173, in __getitem__
    datas = self._table.read(self._cr, self._uid, ids, map(lambda x: x[0],
ffields), context=self._context, load="_classic_write")
  File "/usr/lib/python2.4/site-packages/tinyerp-server/osv/orm.py", line
821, in read
    result =  self._read_flat(cr, user, select, fields, context, load)
  File "/usr/lib/python2.4/site-packages/tinyerp-server/osv/orm.py", line
852, in _read_flat
    cr.execute('select %s from \"%s\" where id in (%s) order by %s' %
(','.join(fields_pre2 + ['id']), self._table, ','.join([str(x) for x in
ids]), self._order))
  File "/usr/lib/python2.4/site-packages/tinyerp-server/sql_db.py", line
75, in execute
    res = self.obj.execute(sql)

I was using the 0.9.7 version

Original issue reported on code.google.com by gunter.k...@gmail.com on 18 Sep 2008 at 12:54

[GoogleCodeExporter]

Have you modified magento_product_synchronize.py? What is on line 87? 

Original comment by nip...@gmail.com on 18 Sep 2008 at 1:55

[GoogleCodeExporter]

product in pool.get('product.product').browse(cr, uid, prod_ids, 
context=context)

select
"magento_tax_class_id","ean13","picture","price_extra","default_code","active","
variants","exportable","calculate_price","product_tmpl_id","price_margin","magen
to_id","product_tmpl_id",id
from "product_product" where id in 
(131072,131073,131074,131075,131076,131077,131078,
131079,131080,131081,131082,131083,131084,131085,131086,131087,131088,131089,131
090,
131091,131092,131093,131094,131095,131096,131097,131098,131099,131100,131101,131
102,
131103,131104,131105,131106,131107,131108,131109,131110,131111,131112,131113,131
114,
131115,131116,131117,131118,131119,131120,131121,131122,131123,131124,131125,131
126,
131127,131128,131129,131130,131131,131132,131133,131134,131135,131136,131137,131
138,
131139,131140,131141,131142,131143,131144,131145,131146,131147,1311 .....

I think the prod_ids array is to big with 20000 ids

Original comment by gunter.k...@gmail.com on 18 Sep 2008 at 2:06

[GoogleCodeExporter]

yeah, right this is definitely a bad code there. Obviously getting such a large 
array
(even only ids) in a single request doesn't sound right to me. Let's find an 
other
way. I don't know if OpenERP supports things like with a Java ResultSet you can
iterate over the records, getting one after an other instead or if we would 
rather
have to persist some state about which product has been the last uploaded. Let 
think
about those alternatives.

Original comment by rva...@gmail.com on 18 Sep 2008 at 8:10

[GoogleCodeExporter]

hey, one solution would be the splitting of the prod_ids array. 

#===============================================================================
# splitting the prod_ids array in subarrays
# ==============================================================================
import math
l=200
f = lambda v, l: [v[i*l:(i+1)*l] for i in 
range(int(math.ceil(len(v)/float(l))))]
split_prod_id_arrays= f(prod_ids,l)
for prod_ids in split_prod_id_arrays:
    ....

In this case, the rest of the code can stay untouched

Original comment by gunter.k...@gmail.com on 19 Sep 2008 at 1:27

[GoogleCodeExporter]

Mmh, I need to figure out how to get notifications on these...

I think there has to be a better way than this. At some point just returning a 
large
number of ids will become a limit, and I think its this kind of thing slowing 
down
the Magento side as well. Instead of getting the ids, then getting the products 
for
the ids, can a call be made to just get the products that are exportable in one 
shot
(assuming the for loop iterates over the results intelligently)

BTW, there is similar code in other places like the sales order import which 
will
need fixing too so it would be good to come up with a general solution.

Original comment by nip...@gmail.com on 19 Sep 2008 at 1:43

[GoogleCodeExporter]

Original comment by GrouhPamart on 26 Sep 2008 at 5:42

• Changed state: Accepted

[GoogleCodeExporter]

Original comment by rva...@gmail.com on 28 Oct 2008 at 10:27

• Changed title: bulk upload of a large product catalog: max_stack_depth too low

[GoogleCodeExporter]

fixed by commit:
http://code.google.com/p/magento-openerp-smile-synchro/source/detail?r=118

Original comment by rva...@gmail.com on 11 Nov 2008 at 1:34

• Changed state: Fixed_on_trunk

[GoogleCodeExporter]

Fixed on last release

Original comment by GrouhPamart on 18 Nov 2008 at 5:46

• Changed state: Verified