Speedtests with Memory Mapped Files in python

[NardJ]

@Photonsters/photonsters I found several options to exchange data between programs on https://en.wikipedia.org/wiki/Inter-process_communication.

I did some tests using Ram Mapped Files in python on an old laptop using a hard drive. I think this simple method could already fast enough for our purposes. For a 100M file, reading takes 0.09 sec, writing also takes 0.09 sec. What do you think?

Below the code I used.

Master.py (reader) `

import mmap
import time

with open("hello.txt", "wb") as f:
    h=b"Hello Python!"
    b=bytearray(100*1000000)
    m=h+b
    f.write(m)

with open("hello.txt", "r+b") as f:
    # memory-map the file, size 0 means whole file
    mm = mmap.mmap(f.fileno(), 0)
    while True:
        mm.seek(0)
        t=time.time()
        ret=mm.readline()  # prints "Hello Python!"
        d=time.time()-t
        print ("Done in ",d," secs.")
        print (ret[:20])
        time.sleep(2)

    # close the map
    mm.close()

`

Slave.py (writer) `

import mmap
import time
with open("hello.txt", "r+b") as f:
    # memory-map the file, size 0 means whole file
    mm = mmap.mmap(f.fileno(), 0)
    mm.seek(0)
    f.seek(0)
    txt="Hello world+" + str(time.time())
    m=txt.encode('ascii')
    h=b"Hello Python+"
    b=bytearray(10*1000000)
    m=h+b

    t=time.time()
    mm.write(m)
    d=time.time()-t
    print ("Done in ",d," secs.")
    print (m[:20])

`

[NardJ]

@Antharon, I tried to make a node.js slave script using mmpa-io, but no success yet. Can you help me with the script?

[NardJ]

I managed to put together node.js versions of slave and master. Python and Node.js code is in the demo folder. All seems to work. Is this the way (or at least an acceptable one) to exchange data between different programs?

[X3msnake]

@bonosoft @Antharon Can you have a look at this?

@NardJ Just out of curiosity how long does it take to do the same from the disk?

[NardJ]

@X3msnake Using 1.1GB data as test case I get the following results:

MMap Writing ... 19.1 secs. Reading ... 0.97 secs.

Disk Writing ... 35.9 secs. Reading ...0.34 secs.