Registry Variables not available during streaming return

[pedersen]

This issue existed on SourceForge. The original can be viewed at https://sourceforge.net/p/turbogears2/tickets/155

[pedersen]

Original Author: kgk, Original Timestamp: 2012-04-18 23:08:34.716000

Original Body: Thread-local variable such tg.request and tg.response are not available to streaming return values in turbogears applications.

i.e. you cannot do

 @expose()
 def mystream():

     yield "boo"
     if tg.request.environ['a'] == 0:
       yield "bah"

     for x in range (1000):
        yield "blah"

During application initialization the RegistrtyManager is created to handle thread local variable i.e. tg.request and tg.response.

TurboGears2-2.1.5: configuration.py:954

app = RegistryManager(app)

Paste-1.7.5.1bisque1-py2.6.egg/paste/registry:367

The RegistryManager includes code to pop registry values after the stream has completed. However you must ask for this behavior with

app = RegistryManager(app, streaming = True)

[pedersen]

Original Author: amolsf, Original Timestamp: 2012-05-02 10:59:01.596000

Original Body: - milestone: 2.2.0 --> Undetermined

[pedersen]

Original Author: amolsf, Original Timestamp: 2012-05-02 10:59:02.099000

Original Body: As it was not one of the targets for 2.2 I'm moving it back to undetermined milestone for evaluation.

We should probably deeply check the side effects of lazy Stacked objects management from Paste RegistryManager as the registry manager expectes the response to always be an iterable he can consume when enabling streaming mode. This might have the side effect of consuming one character at time when returning the html response slowing down the application depending on what the TGApp returns.

[pedersen]

Original Author: krispk, Original Timestamp: 2012-05-02 18:37:17.594000

Original Body: Would this be as simple as checking the return type i.e. string or has an iter attribute and processing in the correct way?

[kgk]

After some debugging, I found that even though you are returning a string in turbogears it is placed into a list or tuple before it gets to paste.. The following patch of paste basically passes strings (templates) unchanged, but allowed streaming apps to access registry variable (after patching turbogears as mentioned above)

BTW it's impossible to post this bug/enhancement to paste.. their bug reporting site has been down for months now. http://trac.pythonpaste.org/pythonpaste/report

Revision: 1504
Branch: default
Author: Kristian Kvilekval <kris@cs.ucsb.edu>  2012-05-08 14:07:36
Committer: Kristian Kvilekval <kris@cs.ucsb.edu>  2012-05-08 14:07:36
Tags: tip
Parent: 1503:8b76085af20c (allow token parameter to be a list or a preformed string)

    Don't penalize streaming applications.. test response if streaming needed.

------------------------------ paste/registry.py ------------------------------
@@ -372,11 +372,18 @@
         app_iter = None
         reg = environ.setdefault('paste.registry', Registry())
         reg.prepare()
-        if self.streaming:
-            return self.streaming_iter(reg, environ, start_response)
+        #if self.streaming:
+        #    return self.streaming_iter(reg, environ, start_response)

         try:
             app_iter = self.application(environ, start_response)
+            #print "REG ", type(app_iter)
+            if isinstance(app_iter, (list, tuple)):
+                #print "DIRECT"
+                return app_iter
+            #print "STREAMING"
+            return self.streaming_iter(app_iter, reg, environ)
+
         except Exception, e:
             # Regardless of if the content is an iterable, generator, list
             # or tuple, we clean-up right now. If its an iterable/generator
@@ -404,9 +411,9 @@

         return app_iter

-    def streaming_iter(self, reg, environ, start_response):
+    def streaming_iter(self, app_iter, reg, environ):
         try:
-            for item in self.application(environ, start_response):
+            for item in app_iter:
                 yield item
         except Exception, e:
             # Regardless of if the content is an iterable, generator, list

[pedersen]

The first question that comes to mind is this: Can we do this without patching Paste? Having our own, locally modified, copy is something that I do not like, and would very much like to avoid doing. Add in that we're not the maintainers for Paste, and this can be difficult to get published.

Is there a way to modify TG to fix this?

[amol-]

a0769d5010a3e9e4faa1d0cbeb2d44299d24770d 75273b221c94fea6af3d8e2f76068de9701525c4 and 29540bed9f22113a88cb0a0118b9f6cc982a0175 fixed this issue, making possible to access tg.request, tg.response and so on during streamed responses.

[kgk]

Thanks for fixing this. What version is this fixed in 2.3?

[amol-]

yes, it's in 2.3