ImageDraw.point speed differences between the PIL 1.1.7 and Pillow 4.0.0 implementations

[DaisenryakuFan]

What did you do?

I've been using the ImageDraw.Point function to individually draw pixels on a very large image (9000x7000 pixels) and there's a significant difference between performance in PIL 1.1.7 and Pillow 4.0.0

It takes about 10~ (in compiler) seconds on Python 2.7 and PIL 1.1.7 (Win32) to do a complete set of drawing routines; but when I ported my code over to Python 3.4 and Pillow 4.00 (Win32); the time increased by 10-fold to 111~ seconds.

I've attached the code and data files necessary to run it (v2_x is for Python 2.7 and v3_x is for Python 3.4)

ProblemIssue.zip

Thanks in advance.

[DaisenryakuFan]

....I'm a idiot. I changed things slightly and got the version 2 "pixel access" method described here that I originally used:

http://effbot.org/zone/pil-pixel-access.htm

working in v3.

It's still a bit slower than in PIL 1.1.7 -- it takes 26~ seconds to dump, versus 10~ seconds; but I can live with that.

I wouldn't have even thought to do the line of attack I used in the attached until I posted here; so this thread wasn't a waste of digital bits.

ProblemIssueSomewhatFixed.zip

[hugovk]

You can profile the code to see where the big differences are, for example:

https://julien.danjou.info/blog/2015/guide-to-python-profiling-cprofile-concrete-case-carbonara

[wiredfool]

I've taken a quick look at this, without running a profiler.

In general, I wouldn't recommend using pixel access for large scale operations, or when you're worried about performance. At the point when you're iterating over the whole image, it's time to look at alternate methods. Given that you're doing something like 60 million pixel access operations, I'm not surprised it's slow. At that scale, small changes to the number or objects created or changes to the the c-api layer would start to be noticed.

So, some thoughts in no particular order.

• A palette image would make a ton of sense here. You're essentially doing a palette image manually, using 4x the memory. You can convert to RGBA before saving if necessary, or just save the png with the small palette.
• You'd be better off consolidating your writes in chunks, so that instead or writing a pixel, you're blitting a whole hexagon at a time. You can do this either by prerendering each color hexagon in scratch space and then pasting it on your large image with ImageDraw.paste, or use ImageDraw.polygon. I'd also recommend adding the black borders at this stage, so that you can start with a blank image rather than a gigantic tiff. This would lead to something like 43k calls into the C layer, rather then 60 million.

[wiredfool]

Ok, back at the KB, using Image.paste and prerendered hexagons on a blank image, I'm seeing a total runtime of ~3.5~ 13 seconds, vs 82 seconds for the original. This is without turning this into a palette image, or dealing with smarter file handling in the outer loop.

(edit -- I was cropping this smaller on output so that it didn't crash display. Save actually takes ~10 seconds, so the actual time breakdown on run was 1.5 sec setup, 2.5 sec render, 10 sec save. Original run was 8 sec setup, 62 second render, and 12 second save)

#Basic File Structure. The AE map is a Hexagon grid with 232 (x) rows by 205 (y) columns.
#The basic file structure is a binary file with 47,560 records (aka hexes). There is no header.
#Each record defines data for an individual hexagon and is 56 bytes wide.
#The starting address of any particular hexagon is ((205*X)+Y)*56
##################################################################################
##################################################################################
import time #This is used to count time.
from PIL import Image #This is the PIL image library.
from v2_MAPGEN_Function_Dump import FillHex #Import FillHex
####################################################################################

######################################################################
## BIG FUCKING NOTE HERE
## THE IMAGE FILE MUST BE 32 BIT FOR THE PIXEL PROCESSING TO WORK!
######################################################################

start1 = time.time() #Start Timer
#orig = Image.open("WITP_AE_Grid_Blank_32Bit.tif") #Open Imagefile.
## pix = im.load() #Load Image into memory.
## end1 = time.time() #End Timer
## print ("LAYER FILE Loaded in "),
## print (end1-start1),
## print (" seconds.")
## print ("The image size is "),
## print im.size, #Get the width and hight of the image for iterating over
## print (" pixels.")

im = Image.new('RGB', (9766,7803), "black") # or orig.size

#########################################
Map_DeepOcean = (0, 0, 192) #Set image color (0x0).
Map_Clear = (192, 255, 192) #Set image color (0x1).
Map_Jungle = (64, 255, 64) #Set image color (0x2).
Map_Mountain = (150, 50, 11) #Set image color (0x3).
Map_Desert = (255, 255, 0) #Set image color (0x4).
Map_Swamp = (0, 128, 128) #Set image color (0x5).
Map_HeavyUrban = (255, 0, 0) #Set image color (0x6).
Map_Forest = (0, 192, 0) #Set image color (0x7).
Map_Rough = (0, 64, 64) #Set image color (0x8).
Map_SandyDesert = (255, 255, 192) #Set image color (0x9).
Map_Tundra = (192, 192, 192) #Set image color (0x0a).
Map_IceField = (225, 225, 225) #Set image color (0x0b).
Map_Atoll = (64, 255, 255) #Set image color (0x0c).
Map_LightUrban = (255, 128, 128) #Set image color (0x0d).
Map_TropicalMountain = (93, 135, 93) #Set image color (0x0e).
Map_RoughDesert = (192, 192, 0) #Set image color (0x0f).
Map_RoughForest = (0, 64, 0) #Set image color (0x10).
Map_RoughJungle = (0, 60, 60) #Set image color (0x11).
Map_ShallowOcean = (192, 255, 255) #Set image color (0x12).
Map_PackIce = (255, 255, 255) #Set image color (0x13).
#########################################
#raw_input("PRESS ENTER TO PROCESS MAP IMAGE TO PWHEXE.DAT") #HOLD
#########################################

def render_tile(color):
    im = Image.new('RGB', (52,52), 'black')
    pix = im.load()
    FillHex(0,0,color,pix)
    return im

tiles = [ render_tile(v) for v in (
    (Map_DeepOcean,
     Map_Clear,    
     Map_Jungle,   
     Map_Mountain, 
     Map_Desert,   
     Map_Swamp,    
     Map_HeavyUrban,
     Map_Forest,    
     Map_Rough,     
     Map_SandyDesert,
     Map_Tundra,     
     Map_IceField,   
     Map_Atoll,      
     Map_LightUrban, 
     Map_TropicalMountain,
     Map_RoughDesert,     
     Map_RoughForest,     
     Map_RoughJungle,     
     Map_ShallowOcean,    
     Map_PackIce))]

mask = render_tile((255,255,255)).convert('1')

start2 = time.time() #Start Timer for HEX DUMPING.

DECIMAL_OFFSET = 0 # Initalize and set decimal offset for file seek to zero.
RECORD_OFFSET = 31 # This determines the byte that will be dumped.
ACTUAL_OFFSET = DECIMAL_OFFSET + RECORD_OFFSET #This will be the one that goes into the Seeker.

HEXID = 0 # Initalize and set Hex ID to zero.

HEXLOC_X = 0 # Init and set X Hex Location to 0.
HEXLOC_Y = 0 # Init and set Y Hex Location to 0.

HEXCOLOR = [0,0,0] #initalize hexcolor RGB triplet.
HEXVALUE = "" #Initalize Hex Value.

fp = open("pwhexe.dat","r+b")   #opens PWHEXE.DAT in read only binary mode.

COUNTER_LOOP = 1 # Defines Loop Variable.
for COUNTER_LOOP in range (1,47560): # Runs 47,560 times as that's how many hexes there are in WITP:AE.
    #################################################
    #######BEGIN LOOPING SEQUENCE####################
    #################################################
    # PSEUDOCODE BELOW
    #Get Decimal Offset.
    #Go to Location, pull data from it.
    #Calculate X,Y Location from Offset.
    #Increment decimal offset by 56 and repeat cycle again until decimal offset 2663304 (final hex)
    ##########################################################

    fp.seek(ACTUAL_OFFSET) # Go to the file offset to start recording.

    ##BEGIN CALCULATING HEX X AND Y COORDINATES FOR MAP DUMP.
    HEXID = DECIMAL_OFFSET/56 # Calculate HEXID
    HEXLOC_Y = HEXID%205 # Calculate Y location
    HEXLOC_X = (HEXID-HEXLOC_Y)/205 # calculate X location
#        print ("Reading Hex "),
#        print (HEXLOC_X),
#        print (","),
#        print (HEXLOC_Y),
#       print (" at offset "),
#       print (DECIMAL_OFFSET)
#       print ("Byte "),
#       print (RECORD_OFFSET),
#       print (" of record.")

    char = fp.read(1)   # Read One Byte from file as a character.
    byte = ord(char) #convert to Character.

    PIXLOC_Y_MODIFIER = (HEXLOC_Y * 38)

    if (HEXLOC_Y % 2 == 0): #even 
        PIXLOC_X_MODIFIER = (HEXLOC_X * 42) #Calculate Pixel Y location.
    else: #odd #21
        PIXLOC_X_MODIFIER = (HEXLOC_X * 42) + 21 #Calculate Pixel Y location.
    ###

    im.paste(tiles[byte], (PIXLOC_X_MODIFIER, PIXLOC_Y_MODIFIER), mask)

    DECIMAL_OFFSET += 56 #Increment Decimal Offset by 56 to move to the next HEX in the .dat file.
    ACTUAL_OFFSET += 56 #Increment Actual Offset by 56 to move to the next HEX in the .dat file.

###############################LOOP IS OVER########################################################################

##ONCE WE ARE ALL DONE....
end2 = time.time() #End Timer
im.save("smaller.png") # Save the modified pixels as png
fp.close()  # Closes PWHEXE.DAT to return control to OS.

print ("PWHEXE.DAT PROCESSED. IT TOOK "),
print (end2-start2),
print (" seconds.")