Cross-device link error from gomobile image processor

[GoogleCodeExporter]

What steps will reproduce the problem?

Images that are embedded within richdocuments are _not_ displayed by gomobile.  
When run in "fg" mode, the log output from my instance contains errors such as 
the following:

- - -
2010-11-03 22:52:16 ERROR Zope.SiteErrorLog 1288839136.540.277141060035 
http://m.toughbook.lan/@@mobile_image_processor
Traceback (innermost last):
  Module ZPublisher.Publish, line 127, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 47, in call_object
  Module gomobile.mobile.browser.imageprocessor, line 636, in __call__
  Module gomobile.mobile.browser.imageprocessor, line 474, in serve
  Module gomobile.mobile.browser.imageprocessor, line 149, in set
  Module gomobile.mobile.browser.imageprocessor, line 136, in closeTempFile
OSError: [Errno 18] Cross-device link
- - -

Initially the image_resize_cache_path setting was "/tmp", my plone instance 
(/opt/plone) resides in a different filesystem mounted on /opt.  I have changed 
the image_resize_cache_path to point to "/opt/tmp", but the errors continue and 
no images are displayed to the web browser.

What version of the product are you using? On what operating system?

I am using mfabrik.webandmobile=0.9.1 plone=4.0.1-unified-installer on 
freebsd=8.1.

Please provide any additional information below.

Details of my product versions are contained in the attached file.

Original issue reported on code.google.com by br...@huron-data.com on 4 Nov 2010 at 3:26

Attachments:

• picked_versions.txt

[GoogleCodeExporter]

Hi Brett,

I will check this issue. It should respect image_resize_cache_path, but if 
there is a problem reading properties, it will fall back to the default

image_resize_cache_path = 
getattr(self.context.portal_properties.mobile_properties, 
"image_resize_cache_path", DEFAULT_CACHE_PATH)

The default is

DEFAULT_CACHE_PATH="/tmp/gomobile_image_cache"

Original comment by mioht...@gmail.com on 4 Nov 2010 at 1:19

• Changed state: Accepted

[GoogleCodeExporter]

Thank you for the quick response!

Here is another detail.  The gomobile_image_cache directory _is_ being created 
wherever I specify in the configuration.  Currently it is being created in 
/opt/tmp.  No gomobile_image_cache directory is being created in the 
DEFAULT_CACHE_PATH location under /tmp.  So it appears that the properties are 
being read.

The error however is still being generated.  Also, there are _never_ any files 
under /opt/tmp/gomobile_image_cache.  For example:

# find /opt/tmp -type d -print
/opt/tmp
/opt/tmp/gomobile_image_cache
/opt/tmp/gomobile_image_cache/33
/opt/tmp/gomobile_image_cache/33/f3
/opt/tmp/gomobile_image_cache/1e
/opt/tmp/gomobile_image_cache/1e/27

# while true; do find /opt/tmp -type f -print; done
(produces no output)

Original comment by br...@huron-data.com on 4 Nov 2010 at 1:34

[GoogleCodeExporter]

Could it be UNIX file permission issue? Your user running Plone (zope user?) 
does not have right to create files? However I see directories being created so 
it should not be the cause...

Original comment by mioht...@gmail.com on 4 Nov 2010 at 1:57

[GoogleCodeExporter]

I double checked the perms.  The /opt/tmp folder has '1777' permissions 
(drwxrwxrwt) just like the standard /tmp folder permissions.

Should the transformed images persist within the gomobile_image_cache folder?  
...or are they immediately deleted?

Original comment by br...@huron-data.com on 4 Nov 2010 at 2:27

[GoogleCodeExporter]

They should persist. The cache is not cleared unless you specifically call a 
flush URL which takes image resizer secret as a password.

This is the violating code:

    def closeTempFile(self, temp, full):
        """ Perform final cache set as atomic FS operation.
        """
        logger.debug("Created image cache file:" + full)
        os.rename(temp, full)

If you put in breakpoint there or enable log level debug you should see more 
info. You will see if the paths are actually in the different partitions.

Here is a suggested workaround (if the partition issue is never resolved):

https://bugs.launchpad.net/phatch/+bug/566136

I might change the function to use shutils.mv instead of os.rename(). There 
will be a hidden performance penalty, but at least it should work.

Original comment by mioht...@gmail.com on 4 Nov 2010 at 3:21

[GoogleCodeExporter]

Mikko,

I turned on "event-log-level = debug".  Gomobile is attempting to create its 
cache on the same filesystem upon which my instance has been installed.  
Everything is under /opt.

# mount
/dev/ad0s1a on / (ufs, local)
devfs on /dev (devfs, local, multilabel)
/dev/ad0s1e on /tmp (ufs, local, soft-updates)
/dev/ad0s1f on /usr (ufs, local, soft-updates)
/dev/ad0s1d on /var (ufs, local, soft-updates)
/dev/ad0s1g on /opt (ufs, local, soft-updates)

I have attached a file with the debug output of a single transaction.  Also 
when event-log-level was still "info" I saw a RuntimeWarning that is probably 
not relevant, but I have attached it as well.

- - -

Is there a method for me to move my existing instance to another directory 
without rebuilding it?  I know many files contain hard-coded paths that would 
need to be updated.

I would like to "mv /opt/plone-4 /plone-4" and see if the problem goes away.  
By default FreeBSD creates /tmp on a separate filesystem, so I still will not 
be able to use the default setting for image_resize_cache_path.  Maybe I should 
mv the entire instance to /tmp and see what happens!

Thanks,
Brett

Original comment by br...@huron-data.com on 4 Nov 2010 at 4:15

Attachments:

• log_level_debug.txt
• tmpnam_err.txt

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

What do you know, Mikko... I found instructions for moving a Plone buildout 
here: http://plone.org/documentation/kb/moving-buildout

I have experimented with the gomobile installation on several filesystem 
layouts.  It turns out that no matter how I set up the filesystems under 
FreeBSD I always get the "Cross-device link" error and never see any images.

On Linux however, the results are very different.  Placing the image cache on 
the same or on a separate filesystem under Linux has no impact on the 
imageprocessor.  It always successfully resizes and displays the images for the 
mobile browser with no complaints.

I have made every attempt to keep the FreeBSD and Linux buildouts identical.  
In both cases the Unified Installer is building a standalone Python2.6 for use 
with Plone.

By the way, I appreciate the work you have done on this and other Plone-related 
projects.  I am very excited to roll out the gomobile to the sites that I 
manage.

Brett

Original comment by br...@huron-data.com on 5 Nov 2010 at 3:09

[GoogleCodeExporter]

Ok. Then I suspect it is a bug in Python. We will implement workaround using 
shutil package as instructed in the bug link above.

Original comment by mioht...@gmail.com on 5 Nov 2010 at 8:54

[GoogleCodeExporter]

If you do a trunk update can you try to install the trunk version on BSD server?

Original comment by mioht...@gmail.com on 5 Nov 2010 at 10:24

[GoogleCodeExporter]

If I do a trunk update can you try to install the trunk version on BSD server?

Original comment by mioht...@gmail.com on 5 Nov 2010 at 10:24

[GoogleCodeExporter]

Sure thing.  I just installed from "trunk" by adding the following to my 
buildout.cfg:
- - -
[buildout]
. . .
extends = 
http://plonegomobile.googlecode.com/svn/gomobile.buildout/gomobile.plone-4.trunk
.cfg

[gomobile]
checkout-protocol = http
- - -

Now my Plone version has regressed from 4.0.1 to 4.0 and imageprocessor.py is 
still using os.rename. 

I need to figure out how to install the _real_ trunk then I will get back to 
you.  Or... maybe I will simply backport the change into my copy.

Original comment by br...@huron-data.com on 5 Nov 2010 at 2:21

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

By the way, the trunk svn instructions at 
http://code.google.com/p/plonegomobile/source/checkout are not working.  

I am also not sure if you have actually made the change to trunk yet.  Judging 
from the recent committed changes I think not.

Original comment by br...@huron-data.com on 5 Nov 2010 at 2:27

[GoogleCodeExporter]

I just pushed the change to the trunk rev #687 after I succesfully passed all 
tests on Linux.

What part of the instructions is broken?

Original comment by mioht...@gmail.com on 5 Nov 2010 at 2:30

[GoogleCodeExporter]

# svn checkout http://plonegomobile.googlecode.com/svn/trunk/ 
plonegomobile-read-only
svn: URL 'http://plonegomobile.googlecode.com/svn/trunk' doesn't exist

Original comment by br...@huron-data.com on 5 Nov 2010 at 2:37

[GoogleCodeExporter]

Mikko,

The change in Rev r687 works great...  ...a one-liner I should have simply made 
the change to my on-disk copy after Comment 5 and shortened this thread!

Thank you for your time.

Brett

Original comment by br...@huron-data.com on 5 Nov 2010 at 2:48

[GoogleCodeExporter]

Fix released http://plone.org/products/web-and-mobile

Original comment by mioht...@gmail.com on 10 Nov 2010 at 12:09

• Changed state: Fixed