GoogleCodeExporter
commented
8 years ago ================
From Brett:
Marty,
We almost certainly want to compress, then Base64 encode the
compressed data. Three reasons:
1) I have a gut (unproven) feeling that gzip might do a better
job compressing the raw data vs the encoded data.
2) That is the order that the iHarder Base64 encoder uses.
3) The resulting Base64 encoding has no XML special characters
that would need to be escaped in the feed.
I prefer gzip over zip or rar for this purpose because gzip does
not do archiving, so uncompressing would not result in the
possibility of producing multiple files.
As far as sending multiple documents in the feed goes, it is my
understanding that the status return simply acknowledges the
receipt of the feed XML file. The feed file is then queued for
future processing. Any failures processing the feed data are
not communicated back to the CM due to the asynchronous
feed model.
From John:
I'm sure there are compression experts running around. Here's a quick test
using the Perl
MIME::Decoder::Base64 module:
Word file: 11028 KB
Base64-encoded, then gzipped: 4097 KB in 1.7 seconds
gzipped, then Base64-encoded: 4126 KB in 0.8 seconds
google-connectors log file: 49874 KB bytes
Base64-encoded, then gzipped: 15870 KB in 10.9 seconds
gzipped, then Base64-encoded: 7538 KB in 3.5 seconds
I used a different Base64 encoder, so the timing is not directly relevant. It
makes sense, though, because
the gzip-first approach touches less data, unless the compression increases the
size by more than a
nominal amount.
If it's Base64-encoded then gzipped, we have to gzip the entire XML envelop of
the feed, where if it's
gzipped then Base64-encoded, that's just a variation on the content element as
you mentioned.
I don't know what technology the feedergate uses, but we should check whether
it supports Content-
Encoding: gzip using Apache/mod_deflate or similar. That may just be even
easier than supporting gzip in
the parser, although it may not perform as well.
I agree with Brett on the feed status. All we get back is that the server
accepted the feed, which isn't
different for multiple documents compared to a single document. I'm OK with
that.
It would be much trickier to break the Base64 into lines just for the
teedFeedFile. It would be doable, but it
would mean more extra scanning of the stream. Can you run that by the GSA team,
too, whether the feed
parser could handle the line breaks? Also, whether it handles arbitrary breaks
or would require 72 or 76
character lines? The goal is just to not have 16 MB lines in a file, so it
might be faster and easier to break
lines when the I/O loop finishes encoding a 2K chunk or whatever size it is.
John L
Original comment by Brett.Mi...@gmail.com on 26 May 2009 at 4:41
- Added labels: Type-Enhancement
Original issue reported on code.google.com by
Brett.Mi...@gmail.comon 26 May 2009 at 4:39