If you restart Zope logged-in users with non-empty carts won't be able to log in again

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Log in (or register)
2. Put something in your cart
3. Restart Zope

What is the expected output? What do you see instead?
The site won't work for pages where the cart portlet is shown and the
following traceback will appear:
...
#  Module Products.CMFPlone.ActionsTool, line 114, in listFilteredActionsFor
Module Products.CMFPlone.ActionsTool, line 83, in listActionInfos
Module Products.CMFCore.ActionInformation, line 200, in __getitem__
Module Products.CMFCore.Expression, line 48, in __call__
Module Products.PageTemplates.ZRPythonExpr, line 49, in __call__
__traceback_info__: path('object/@@getpaid_control').showManageCart()
Module PythonExpr, line 1, in <expression>
Module Products.PloneGetPaid.browser.content, line 304, in showManageCart
Module Products.PloneGetPaid.cart, line 48, in get
Module Products.PloneGetPaid.cart, line 55, in _getCartForUser
Module zope.app.container.ordered, line 133, in __len__
Module ZODB.Connection, line 761, in setstate
Module ZODB.Connection, line 801, in _setstate
Module tempstorage.TemporaryStorage, line 106, in load

Please use labels and text to provide additional information.
I guess this has to do with the temporary cart storage.

Original issue reported on code.google.com by silviot on 1 Sep 2008 at 3:38

[GoogleCodeExporter]

leonelgo... after applying persistant-safe-cart.patch did you clean out all the
getpaid data? Perhaps that error is being generated by a return shopper?

Original comment by michael....@gmail.com on 13 Jul 2009 at 8:07

[GoogleCodeExporter]

Attached is a patch for ZODB that prevents cross database references from being
created in the first place. While it won't pinpoint the exact place that such a
reference is being created it will pinpoint the exact request responsible for 
the
cross reference. And prevent it from succeeding.

You'll see an error along the form:

2009-07-13 19:22:33 ERROR Zope.SiteErrorLog
http://localhost:8080/plone/@@getpaid-checkout-wizard
Traceback (innermost last):
  Module ZPublisher.Publish, line 125, in publish
  Module Zope2.App.startup, line 238, in commit
  Module transaction._manager, line 96, in commit
  Module transaction._transaction, line 395, in commit
  Module transaction._transaction, line 495, in _commitResources
  Module ZODB.Connection, line 510, in commit
  Module ZODB.Connection, line 555, in _commit
  Module ZODB.Connection, line 582, in _store_objects
  Module ZODB.serialize, line 408, in serialize
  Module ZODB.serialize, line 417, in _dump
  Module ZODB.serialize, line 327, in persistent_id
InvalidObjectReference: Don't allow implicit cross-database references

Original comment by michael....@gmail.com on 13 Jul 2009 at 8:07

Attachments:

• dont-allow-xrefs.patch

[GoogleCodeExporter]

lol lol. Sorry, leonel was me again... Once again I forgot to logoff and another
friend had used the same computer I use...I will pay more attention to this...

Thanks michael for all your help. 

I don't use any shipping method, but my products are all shippable itens. I did 
that
because my Pagseguro calculates the shipping cost for me, as far as I send in 
the
form the weight of the product. So I use shippable itens, because they have 
weight
attributes, but only for that...

  The error is much less frequent after persistant-safe-cart.patch , so one of the
causes was definetively fixed. I think this should be uploaded to SVN.

   I have disabled the return from the shopper some time ago, so i doubt this is the
problem.

  I applied the patch clothed-exceptions.patch to get more information, but I think
something may be wrong. One someone clicks on add to cart I get this line in 
the log:

2009-07-13T11:47:52 INFO Products.PloneGetPaid try: sessions.set_came_from_url

Is this correct? nothing else besides that... I imagine it should return as a 
string
the URL, not this line, am I correct?

I am not sure if I should apply dont-allow-xrefs.patch because to pack I 
upgraded my
ZODB to 3.8.1. Please tell me if I should...

Is anyone else battling against this issue? If it is just me, maybe michael and
others trying to help should not waste theirs time on this...

Original comment by rafaelcr...@gmail.com on 13 Jul 2009 at 4:54

[GoogleCodeExporter]

Hi again,

I think I know why the error is less frequent now. 
As I said in comment #13, I was getting a zodb corrupted even if I did not 
restart
zope with logged users. Before I got it even without restarting the zope at 
all...
persistant-safe-cart.patch seems to have fixed this most worst behaviour .

Now I am just getting the error when I restart zope with logged users with non 
empty
carts... Back to the original error... This I can discipline myself and others 
using
the site as registered users not to happen.. So I can live with that... (If my
hipothesis is correct)

thanks

rafael

Original comment by rafaelcr...@gmail.com on 13 Jul 2009 at 5:13

[GoogleCodeExporter]

Hi Rafael,

The patch dont-allow-xrefs.patch can be applied to ZODB 3.8.1. Yes please apply 
that
patch.

Sounds like you have applied clothed-exceptions.patch just fine. Also sounds 
like
that is not reporting any problems. Those "try: sessions.set_came_from_url" 
messages
are my mistake - I meant to remove those particular messages from the patch - 
though
they are harmless. That patch is intended to log error details "except:
sessions.set_came_from_url" if a problem is found.

Original comment by michael....@gmail.com on 13 Jul 2009 at 8:43

[GoogleCodeExporter]

Rafael,

I wonder if you have a corrupt intid registry. I don't think that gets 
reinstalled
when you reinstall getpaid. Iterate through the intid registry and see if you 
get any
errors. You should be able to do that using:

  bin/instance debug

and something along the lines of:

  >>> app.plone
  <PloneSite at /plone>
  >>> from five.intid import site
  >>> for uid, key in site.get_intids(app.plone).items():
  ...     print uid, key, key.path, key.dbname, key.object
  ...
  582905855 <five.intid.keyreference.KeyReferenceToPersistent object at
  [SNIP endless output]

Original comment by michael....@gmail.com on 15 Jul 2009 at 3:55

[GoogleCodeExporter]

Hi Michael,

Sorry for taking too long to answer. I was out of the town for two days.
You are right, I have a corrupt intid registry... Is it possible to fix it? 
Attached
the output I got iterating through it...

Original comment by rafaelcr...@gmail.com on 17 Jul 2009 at 1:00

Attachments:

• outpu.txt

[GoogleCodeExporter]

Hi Michael,

Reading again my output, maybe everything is ok?
 Some time ago. I moved my online store to its own site (cutted and pasted the
products, uninstalled getpaid from the former and reinstall in the new) to get 
it
isolated from the rest of my sites.Should I iterate through the other Plone 
Sites I
have in the same instance? 
 I just applied dont-allow-xrefs.patch. Until now, nothing raised... As Soon as I get
something I will post here...

Thanks again...

Original comment by rafaelcr...@gmail.com on 17 Jul 2009 at 1:25

[GoogleCodeExporter]

Hi again,

I have already had to reinstall getpaid once after my last post...
dont-allow-xrefs.patch has produced no error? I am wondering why... Maybe the
exception was not raised because that line was not reached? So maybe the 
corruption
is not due to cross reference?
I am using getpaid.pagseguro as a payment processor. it is basically a copy of
getpaid.paypal... Maybe the error is there? Is anyone else getting this error?
Because only me am posting with it ...
Maybe I should start a new Data.fs and import zexp of my fodlers? would this be 
a
good idea?

Rafael

Original comment by rafaelcr...@gmail.com on 18 Jul 2009 at 9:05

[GoogleCodeExporter]

In getpaid.pagseguro, after saving the button with the html form to be sent to
Pagseguro server, a cart is destroyed:

 # save html for button - we'll destroy the cart later on
        html = button.cart_post_button(order)

        # and destroy the cart
        cart_util.destroy(self.context)

Should those lines be fixed with load dumps trick?

Thanks a lot,
Rafael

Original comment by rafaelcr...@gmail.com on 18 Jul 2009 at 9:10

[GoogleCodeExporter]

As I ahve mentioned before, installing zodb 3.8.1 allowed me to pack. Today i 
tried
to update to 3.3rc4 and got an error. Even uninstalled plonegetpaid, I can't
migrate... Maybe this can help sort out the origin of the problem? Attached the 
traceback

Original comment by rafaelcr...@gmail.com on 18 Jul 2009 at 10:43

Attachments:

• migration_failed.txt

[GoogleCodeExporter]

That dont-allow-xrefs.patch has not yet produced an error suggests that either 
the
problematic cross reference was created before the patch was applied. Or as you 
said
- the remaining problem is not due to a cross reference.

Your output from the intid inspection looks too short. It only has one object. 
That
one object looks just fine though. I was expecting significantly more objects 
than
that. Maybe you want to remove "key.object" from the output for the first run. 
Or use
"repr(key.object)" instead.

Just reading the code sample in #60 - looks okay. I'm guessing that html is a 
simple
string.

That problem with migration looks like progress. You can now recreate the 
problem (or
a variation of it) on demand. From the traceback that again looks like it's 
some kind
of corruption within the intid registry. A starting point would be to find out 
what
'ob' is just before five/intid/intid.py", line 33 is called.

Original comment by michael....@gmail.com on 19 Jul 2009 at 8:04

[GoogleCodeExporter]

I just noticed that if you are using getpaid.formgen and the one page checkout, 
it
looks like it will store orders with dangling remote references to shopping 
carts. 
i'm going to update this code:

getpaid/formgen/content/checkout.py line 90
        shopping_cart = oneshot
    if oneshot is None:
            #If this is not to be a disposable cart transaction we get the cart as we
should.                      
            #NOTICE: This has not been tested since there is no process currently
that uses it                     
            shopping_cart = component.getUtility( interfaces.IShoppingCartUtility
).get( self.context )
            shopping_cart = loads( dumps( shopping_cart ) )

so the loads(dumps()) is not nested under the if:

Original comment by larubbio@gmail.com on 31 Jul 2009 at 11:14

[GoogleCodeExporter]

Hello All.

I've also experienced this. I have a demo instance up 

(http://plonedemoshop.no-ip.biz/)

and whilst I was testing / debugging I had this happen a number of times, 
except 
that in my case, the cart portlet was throwing an exception.

My setup is:
(Zope 2.10.6-final, python 2.4.5, linux2)   
2.4.5 (#1, Sep 6 2009, 11:40:59) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)]
    *  Plone 3.1.7
    * CMF 2.1.1
    * Zope (Zope 2.10.6-final, python 2.4.5, linux2)
    * Python 2.4.5 (#1, Sep 6 2009, 11:40:59) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)]
    * PIL 1.1.6

Initially, the only fix I could find was to do a physical DB file deletion. 
Later I
discovered that deleting the .installed.cfg and doing bin/buildout -N seems to 
fix
the problem. However I have been unable to determine why this happens. Catalog
clearing / reindesxing etc seems to have no effect either. 

I surmise that the zope db is holding a reference to an incomplete transaction, 
other
than that, this bug baffles me!!

RM. 

Original comment by RomanOrb...@gmail.com on 8 Oct 2009 at 2:54

[GoogleCodeExporter]

Hi RomanOrbiter,

What you mean by a phisical db delete? Inspecting data.fs and deleting the 
objects
corrupted? How you do this? The solution I have (and I must do that every day 
or so)
is after the bug comes out reinstall getpaid through portal_quick installer.

What does installed.cfg has inside it? 

Thanks,
Rafael

Original comment by rafaelcr...@gmail.com on 9 Oct 2009 at 2:37

[GoogleCodeExporter]

Hope this helps - at least it does work for me...

------------------------------8<-schnipp-------------------------------
import transaction
from zope.app.component.hooks import setSite
from five.intid.site import add_intids, del_intids

def login(app, username='admin'):
    app = setupDummyRequest(app)
    user = app.acl_users.getUserById(username).__of__(app.acl_users)
    newSecurityManager(None, user)
    return app

app = login(app, 'admin')
portal = app.plone_instance
setSite(portal)
del_intids(portal)
add_intids(portal)
transaction.commit()
app._p_jar.close()
------------------------------8<-schnapp-------------------------------

After that you might have some broken references, which leads to the unability 
to pack your ZODB if you are 
running ZODB<3.8.1. Pinning the ZODB egg to ZODB===3.8.1 lets you pack again.

As said before, this works for me, YMMV.

Cheers,

Andreas

Original comment by d1sk0k41s3r@googlemail.com on 9 Oct 2009 at 3:38

[GoogleCodeExporter]

Hi, Trying to run the script in 66, I get 

Traceback (most recent call last):
  File "<string>", line 1, in ?
  File "script", line 12, in ?
    app = login(app, 'admin')
  File "script", line 7, in login
    app = setupDummyRequest(app)
NameError: global name 'setupDummyRequest' is not defined

From where should I import this method?

Thanks,

Rafael

Original comment by rafaelcr...@gmail.com on 4 Nov 2009 at 8:37

[GoogleCodeExporter]

the script in 66 is incomplete and needs to be extended with something like 
this:

------------------------------8<-schnipp-------------------------------
from AccessControl.SecurityManagement import newSecurityManager
from ZPublisher.BaseRequest import RequestContainer
from ZPublisher.Request import Request
from ZPublisher.Response import Response

def setupDummyRequest(app):
    response = Response()
    environ = {}
    environ['SERVER_NAME'] = 'nohost'
    environ['SERVER_PORT'] = '80'
    environ['REQUEST_METHOD'] = 'GET'
    request = Request(sys.stdin, environ, response)
    request._steps = ['noobject']
    request['ACTUAL_URL'] = request.get('URL')
    return app.__of__(RequestContainer(REQUEST=request))
------------------------------8<-schnapp-------------------------------

HTH,

Andreas

Original comment by d1sk0k41s3r@googlemail.com on 4 Nov 2009 at 10:13

[GoogleCodeExporter]

I have applied the patch by michael.dunstan (see comment 36). This does seem to 
resolve the source of the 
error, which makes sense - i.e. to do a deep copy of the cart when transferring 
it from anonymous session to 
the persistent sessions in the ShoppingCartUtility. This was most easily 
reproduced by starting out as an 
anonymous shopper, put something in the cart, log in so you trip the transfer 
of the cart to the persistent 
storage, and then restart zope and click somewhere in your site the cart turns 
up while still logged in, or log 
in again if you have to.

However, if you still have users that have active carts, you will still see the 
problem - the data you get back 
for the user from _sessions is corrupt - presumably because it contains 
pointers to now non existent objects 
that were in the temporary storage session manager. In these cases I have used 
a try and except around 
http://code.google.com/p/getpaid/source/browse/Products.PloneGetPaid/trunk/Produ
cts/PloneGetPaid/cart.
py?spec=svn3225&r=3225#71 to test the state of the cart object returned and if 
it's corrupt to create an 
entirely new one.

Let me know if you have these errors still and perhaps I can help you get out 
of the hole.

Original comment by matt.hal...@gmail.com on 13 Nov 2009 at 7:16

[GoogleCodeExporter]

  Hi, I will test it next weekend.. What is working now is having my client run
kaiser's script 1 min after the daily cron restart... I can now sleep a little
better. No sideeffect due to deleting intid every day. But I am hopeful that 
matt's
patch can fix it forever...

Original comment by jovemcom...@gmail.com on 17 Nov 2009 at 1:59

[GoogleCodeExporter]

This bug doesn't appear anymore when running with Plone 4, the cart contents are
retained after restarts

Original comment by alberto....@gmail.com on 25 Mar 2010 at 3:04

[GoogleCodeExporter]

@alberto.berti are cart contents persistent between restarts for anonymous 
users too?

Original comment by silviot on 25 Mar 2010 at 3:13

[GoogleCodeExporter]

No they are not. And this is expected behaviour (in a way not a bug) since 
sessions
are cleared in Zope on a restart. I suggest using persistent sessions, 
described here: 

http://longsleep.org/1/howto/sharesessionwithzeo (down at the moment though)

and 

https://bugs.launchpad.net/collective.buildout/+bug/427878

A related problem is the use of load balancing. The session data used to be pr.
zeo-client. ( I guess it still is). This means you can have a cart disappear and
reappear at random. So a way of making the sessions sticky, either by making 
sure
requests from the same browser (in the same session) gets to the same 
zeo-client, or
by making the session data persist in the zeo server. Also descibed in the 
above links.

Original comment by sune.woeller@gmail.com on 25 Mar 2010 at 3:59

[GoogleCodeExporter]

Ah, the above was straight from memory - it applies only to persisting the 
session
data in zeo.

Original comment by sune.woeller@gmail.com on 25 Mar 2010 at 4:06

[GoogleCodeExporter]

for reference: check section 21.7.4 in
http://docs.zope.org/zope2/zope2book/Sessions.html - "If you want your session 
data
to persist across server reboots"...

Original comment by sune.woeller@gmail.com on 25 Mar 2010 at 4:13

[GoogleCodeExporter]

Is there a final solution to get rid of those KeyErrors problems (comment 33)?
I'm having this problem each time I restart the server and I'm forced to use a

...
del_intids(portal)
add_intids(portal)
...

which causes other problems in the portal ( it breaks the mailing lists of 
singing &
dancing...)

Thank you

Original comment by i...@martronic.ch on 30 Mar 2010 at 8:28

[GoogleCodeExporter]

Ok, the problem seems to be coming from the Transient Object Container which is
contained in a Temporary Folder. That means that session data are deleted as 
zope
restarts.
The documentation mentionned in post 75 gives no solution to store the session 
data
permanently and I haven't found recent doc about it.

So for the moment I just copied the Transient Object Container located in the 
root of
zope /temp_folder/session_data directly into to the root /session_data and 
change the
path in the session_data_manager. That make session data persists.

Everything seems to be working fine now, no more KeyErrors even with zope 
restarts.
But I don't know if this manipulation has side effects...

I hope this will be fixed in further versions of Zope.

Original comment by i...@martronic.ch on 3 Apr 2010 at 8:14

[GoogleCodeExporter]

@info@martronic.ch: what zope version are you using?

Original comment by sune.woeller@gmail.com on 28 Apr 2010 at 1:03

[GoogleCodeExporter]

hi everybody 

To summarize:

There have been two distinct problems in this ticket:

a) the creation of references to items in temporary storage (the session) - 
solved by
the loads/dumps dance - but there can be more of those problems in third party
packages (getpaid.formgen solved?). Always make sure no reference is made to 
objects
in the session from other persistent objects. The solution is a little messy, 
but it
works. But care is needed when adding stuff to the code. 

b) A corrupt five.intid registry - also referencing objects in temp storage 
that are
no longer there after a restart. This was caused by five.intid registering 
events for
Persistent. Solved by not loading the subscriber.zcml of five.intid, as 
PloneGetPaid
correctly does not, but register own subscribers (for
Products.CMFCore.interfaces.IDynamicType).

See
http://groups.google.com/group/getpaid-dev/browse_thread/thread/4927b9c490b31ac6
/d363eb324d533a66?lnk=gst&q=five.intid#d363eb324d533a66

and for the solution getpaid uses now:
http://groups.google.com/group/getpaid-dev/browse_thread/thread/37856f0246f77a0e
/275ed9f95882a12b?lnk=gst&q=intid#275ed9f95882a12b

Avoiding subscriber.zcml of five.intid is also the solution used by 
plone.app.intid.
But this solution is fragile - put collective.singing/dancing in to the mix (or 
any
other package loading the configure.zcml of five.intid) and the subscribers for
Persistent are active again, and we start registering intids for LineItems etc.
stored in temp storage. After a restart it breaks the whole site as mentioned 
above.

I think this should be solved in five.intid - it should not activate 
subscribers for
Persistent by default. But for now we now the cause of the problem. I'll talk 
with a
S&D developer tomorrow to see if we can make S&D not load the default five.intid
subscribers, but thats a workaround, and there might be other packages loading
five.intid.

Se also this thread:
http://groups.google.com/group/dexterity-development/browse_thread/thread/d8170a
0c90f29136/f835dfe6e75b06f2?#f835dfe6e75b06f2

happy ploning - Sune W.

Original comment by sune.woeller@gmail.com on 28 Apr 2010 at 11:56

[GoogleCodeExporter]

A fix has now been committed to collective.dancing, see here:
http://dev.plone.org/collective/changeset/116425/collective.dancing/trunk

This means the five.intid subscribers for Persistent are no longer loaded. 
Situation
b) from above should no longer happen with S&D trunk.

But be careful since other packages might load the ful zcml of five.intid.

Original comment by sune.woeller@gmail.com on 29 Apr 2010 at 1:50

[GoogleCodeExporter]

With five.intid 0.4.4-1 it's not enough to 

del_intids(portal)
add_intids(portal)

to delete a corrupt intid registry, since five.intid has been made more robust 
to
reinstalls (now the registry is kept, which is a good thing).

I put this in a browser view:

from Acquisition import aq_base
from zope.component import getMultiAdapter
from Products.Five.browser import BrowserView
from five.intid.site import add_intids, del_intids

class GetpaidFixesView(BrowserView):
    ""

    def __call__(self):
        portal_state = getMultiAdapter(
            (self.context, self.request), name="plone_portal_state")
        portal = portal_state.portal()

        del_intids(portal)
        sm = portal.getSiteManager()
        obj = getattr(aq_base(sm), 'intids', None)
        sm._delObject('intids')
        add_intids(portal)
        return 'success'

beware, that it deletes the whole registry and will mess up you S&D 
subscriptions.

The cause of the problem should be fixed on S&D trunk, and PloneGetPaid trunk.

Original comment by sune.woeller@gmail.com on 29 Apr 2010 at 2:36