File not sent immediately but first cached into memory

[GoogleCodeExporter]

Hi,

I don't know this is bug or my fault. Here's the case:

I use environ's file_wrapper to send file to client. It's not file from disk, 
but PCAP file generated dynamicaly from DB and it can be really big. But 
generated file is generated whole into memory and then sent.

I'm stuck on this for last two days and I don't know, what else try. Thoughts I 
had and tried:
1) Content-Length and BINARY data
    - tried to send fake Content-Length, didn't help
2) Tried sending different packeges (package -> one filelike.read() call) with 
STRING as returned value
    - this is interesting:
        - when i tried to send same string (and same length) for each read() call, file was streamed and not buffered
        - when i tried to send random generated string (and random length), file was buffered first and then sent

But... everytime i send generated PCAP file it is buffered. Is this because of 
sending binary data?

I would really appreciate some some help or where to start.

========= MY CODE
=== Returning PCAP - do not work

import cStringIO
from module.netstat.pcap import pcapHeader, pktHeader

class PcapGenerator(object):
    def __init__(self, pool, conn, cursor, size=8192):
        self.size = size
        self.packets = 200

        self.pool = pool
        self.conn = conn
        self.cursor = cursor

        self.buffer = cStringIO.StringIO()
        self.queue = []

        pcapHeader(self.buffer)

    def read(self, size=None):
        size = self.size
        packets = self.packets
        buffer = self.buffer
        queue = self.queue

        # load new packets if needed
        if len(queue) <= 1:
            if len(queue) == 1:
                last = queue.pop()
                buffer.write(last)

            # load from db, build each packet and write to buffer
            rows = self.cursor.fetchmany(packets)
            for row in rows:
                pktHeader(buffer, row[0], row[1], len(row[3]), row[2])
                buffer.write(str(row[3]))

            # read buffer by specified size (tried because returning fixed length strings worked) and prepare values to return
            buffer.seek(0)
            while True:
                data = buffer.read(size)
                if not data:
                    break

                queue.insert(0, data)
            buffer.truncate(0)

        if len(queue) > 0:
            print "RETURN SIZE: ", len(queue[-1])
            return queue.pop()
        else:
            return ''

    def close(self):
        pass

============
=== Returning fixed length string worked
class StringGenerator(object):
    def __init__(self):
        self.loops = 1000000

    def read(self, size=None):
        self.loops -= 1
        if self.loops > 0:
            return 'ABCDEFGHIJKLMNOPGRSTUVWXYZ'

        return ''

    def close(self):
        pass

Original issue reported on code.google.com by neonn...@gmail.com on 19 Oct 2010 at 8:56

[GoogleCodeExporter]

Please repost on the mod_wsgi mailing list. The issue tracker is not for 
getting general help on using or debugging issues with mod_wsgi. 

Original comment by Graham.Dumpleton@gmail.com on 19 Oct 2010 at 9:23

[GoogleCodeExporter]

Discussion shifted to mod_wsgi mailing list. See:

http://groups.google.com/group/modwsgi/browse_frm/thread/ada3632bf3c2912f

Original comment by Graham.Dumpleton@gmail.com on 19 Oct 2010 at 11:29

• Changed state: Invalid
• Added labels: Type-Other
• Removed labels: Type-Defect