Up-to-dated view server

[GoogleCodeExporter]

This is port of javascript view server for couchdb server 0.10+
Support of shows, lists, filters and validate_doc_updates for pythonic views 
included.

Due to currently there is no couchdb-python API to manipulate this functions n, 
I didn't wrote any tests, but it passed all tutorial examples.

Original issue reported on code.google.com by kxepal on 16 Aug 2010 at 2:19

[GoogleCodeExporter]

Did you write this yourself? Is it a port of the JavaScript view server, or did 
you find some documentation on the view protocol?

I definitely want an updated view server, but I prefer that we only add 
something for which we have good tests, so that we can prevent it from going 
stale again. Would you be in a position to write tests for this? Some design 
notes on the protocol would likely also be helpful.

If we decide to take this, I also have some code/style nits.

Original comment by djc.ochtman on 16 Aug 2010 at 2:37

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

It's javascript view server ported by me. I've going by only one thing - to 
make view server maximum compatible with js one, because it's always 
up-to-dated and have released first. So as close pythonic view server will be 
to it so easy support will be.

Shame on me, I've thought for some unknown reason that view server must be 
tested via couchdb-python API - just have look at tests/view.py. I'll make them 
for tomorrow. 

Original comment by kxepal on 16 Aug 2010 at 2:59

[GoogleCodeExporter]

I've found a little problems due passing official view server tests, but now 
all fine.

I'll make better exceptions handling + crush tests later - I really dont like 
this forest of try..except, but couldnt invent something better now - need some 
time to think about it. So this is just check point(:

In additional to javascript view-server features I've implemented two behaviors:
- sealed document: changing document in map function makes no sense for other 
map functions within single view.
- any pythonic exception will crush view server while javascript view server 
allow crushing only on fatal errors - I dont know which ones python have. 
Everything else seems same.

Original comment by kxepal on 19 Aug 2010 at 1:38

[GoogleCodeExporter]

oops, forgot to remove debug handler from view server.

Original comment by kxepal on 19 Aug 2010 at 1:42

[GoogleCodeExporter]

Ok, here is updated view server. Major changes:
- support any couchdb server since 0.9 version. By default view server works in 
mode of compatibility with latest couchdb server version, just run it with 
--couchdb-version key, e.g. view.py --couchdb-version=0.10, to make it work 
well for 0.10 couchdb server. 
- Tests are included for each supported version - all of them had been ported 
from javascript view server.
- Assertion error within validate_doc_update function doesn't count as Fatal 
like any other pythonic exception and will be wrapped as Forbidden error
- "Reduce output must shirnk more rapidly" error now may be occured
- More verbose debug logging

Also I have to add function versionizing decorator to split their behavior for 
each couchdb version. I thinks it will be useful in future to keep legacy 
support without serious code rewriting.

Original comment by kxepal on 3 Sep 2010 at 11:31

[GoogleCodeExporter]

Changes:
- fixed compatibility for python-2.4+
- added more verbose debug output

I've removed old attachments, because they are not actual for now.

Original comment by kxepal on 13 Sep 2010 at 12:09

Attachments:

• view.py
• util.py
• test.py
• testutil.py

[GoogleCodeExporter]

I'm going to need to find a solid chunk of time to review this, which might 
take a while, but I'd really like to take this for the next release...

Original comment by djc.ochtman on 19 Sep 2010 at 1:16

[GoogleCodeExporter]

Issue 140 has been merged into this issue.

Original comment by djc.ochtman on 22 Dec 2010 at 9:13

[GoogleCodeExporter]

I've fixed issue #163 in this view server 
http://code.google.com/r/kxepal-couchdb-python-featured/source/detail?r=e1448890
d2223e0321f96459a53c1757dc5b9662 (just have not seen any reasons to attach once 
again all files for several small changes)

In over way it's ready for 1.0.2 since main change in js view server was in 
sealing documents for map func, but this feature is done already.

I will port sofa and tapirwiki for this view server within next two weeks, so 
this made a great challenge for him and may be more fixed will come if I've 
missed something.

Original comment by kxepal on 12 Feb 2011 at 7:08

[GoogleCodeExporter]

I've found one thing missed - require function to have some modular application 
within document design. But there are some questions about it:
- should it work as in javascript view-server: wrap some abstract code and 
return export values? Would be better to implement python-like import?
- should it support eggs? I think it should, but I have no idea how to import 
eggs inline without saving them on disk. This could be a problem for 
application hosters.
any other behavior suggested?

Original comment by kxepal on 22 Mar 2011 at 11:06

[GoogleCodeExporter]

What I'm doing to handle some efficiency issues with import:

def fun():
  import datetime
  ... other imports ...
  def fun(doc):
    ...
    delta = (datetime.strptime(doc[...]) - datetime.strptime(doc[...])).days
    ...
  return fun
fun = fun()

What really ought to happen is that the view server should go through each 
variable in the exec's locals and check the __module__. If __module__ exists, 
but is None, and the variable points to a callable, then use that as the 
map/reduce func, and error out if more than one is found. This would be 
backwards compatible with existing view functions, but would make it so 
closures are not necessary.

I think the eggs deployment issue has been tackled many times before. Importing 
eggs inline would obliterate responsiveness. Does couch time you out if you 
take to long? I'd be concerned that it would/should. If an application hoster 
supports python, then sooner or later they'd need to come up with a solution to 
handle 3rd party software since, frankly, the power of using python as a view 
server doesn't just lie in "it looks nice" and "it has yield".

To meet couch's same-code-same-result requirements (no side effects), we would 
maybe have to mark imports with python and module version strings, and push 
that back into couch. For example, 'import mythirdpartyegg' would then append 
'#mythirdpartegg py cpython-2.6.5 mod 3.11r7112' to the end of the eval string, 
one per unique detected module. Any time you do any module upgrades, you can 
just delete the version-marking comments out of the view func manually, and 
couch will regenerate it.

However, most modules *tend* to be stable enough API-wise that this isn't a 
problem. If there were any behavior altering bugs/changes to the code, an 
administrator could achieve this manually.

Original comment by extempor...@gmail.com on 22 Mar 2011 at 3:30

[GoogleCodeExporter]

Several questions, could I?
1) what the reason of such wrapper against:
def fun(doc):
  ...
  delta = (datetime.strptime(doc[...]) - datetime.strptime(doc[...])).days
  ...
import datetime

this is not very pythonic to place imports below, but datetime will be tried to 
import only once. However, this style could produce another problem: views must 
have not any state and any dependence from source which could be changed later.

2) Hoster could provide 3rd party modules, but it couldn't provide all versions 
of each template engine, for example. May be you needs trunk jinja2 with you 
own patches, who knows? So idea to create fully portable pythonic couchapp will 
be failed.

3) Is preprocessors statements really good idea? I saw them in couchapp, but 
they have been used only for declaration, not within document design nor view 
server.

As intermediate result, this is implementation of require function as is it 
works for javascript view server:
http://code.google.com/r/kxepal-couchdb-python-featured/source/detail?r=0b6625db
473e74a83df7e9a339899a3c318f7b80

I still need to finish some details, so attachments with new version of view 
server will be later. Sorry, Dirkjan, it seems you to have revise it once 
again, but I'll include documentation for each vital function and more tests to 
make process more easy just(:

Original comment by kxepal on 23 Mar 2011 at 8:45

[GoogleCodeExporter]

You *could* put imports after the inner func that needs them, and python's 
scoping rules would resolve variable lookups, but I agree, it's not Pythonic. 
Closures themselves are not very Pythonic, either. Anyways, why not put imports 
first, as per my suggestion?

The whole reason for using a closure is to avoid the performance penalty of 
repeating the imports for each document. If a function needed k imports and 
there are n documents in a couch database, then, compared to the closure 
technique shown above, there'd be n*k redundant imports taking place, which is 
very slow (python doesn't re-import the module, but there is overhead involved, 
which can be significant).

See: 
http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Import_Statement_Overhea
d

I disagree with you on the severity of Couch's "no side-effects" requirement 
for view/show/list functions. It's a matter of practicality, not a matter of 
theory. Yes, Couch says that the same document passed to the same code must 
produce the same output, no matter how many times it's executed. If the 
document doesn't change, then the result doesn't change.

However, this mandate is only for data correctness. If the module you're 
importing in a view function gets upgraded, and its behavior changes, but your 
view function stays the same (so couch doesn't regenerate the view), then all 
that happens is that your view will be incorrect. Couch won't break (couch 
won't know, mind, or even care).

Also, using module imports don't count as "side effects" at all. Aside from the 
random module, almost all modules (including 3rd party ones) are stateless in 
their behavior. For example, it'd be fine to describe a shape as a list of 
points in a couch document, and then use PIL to draw that to a png image in a 
show function, or to use couch to store server access logs, and then use 
pychart from a list function to generate an svg rendered line graph of server 
traffic.

Furthermore, the above "same code, same doc, same result" requirement does not 
apply across all time. For example, I could define a view function that 
imported some module, and then change the behavior of the module. All I'd have 
to do fix the consistency of the view would be to add a single space to the end 
of the view function, save the design document to couch, and then remove that 
space, and save the design document again. The code is *exactly* the same as 
before, yet we side-stepped the upgrade-changes-behavior issue, and caused 
couch to regenerate its views to reflect the new behavior.  And this process 
can be automated (alternately, you could delete the views from the design doc, 
do a view cleanup, and then reupload the original design document, which would 
require only one view regeneration instead of two). As long as you regenerate 
your views when needed, the "could be changed later" issue isn't a problem at 
all.

In general, hosting vendors are not ever going to support all versions of all 
potential packages. They're either going to support only a handful of popular 
modules (probably Django templating) and force long release cycles, or they'll 
provide you with a few megabytes of space to upload your own modules in your 
own private module path, or they'll not allow the use of any kind of 3rd party 
modules (in which case you might as well use Javascript).

Original comment by extempor...@gmail.com on 24 Mar 2011 at 6:27

[GoogleCodeExporter]

At first till I don't forget, thanks you for detail reply(:

So about imports: 
Allow to have them on top of design function as PEP told us I see is ok too and 
this is much more intuitive behavior.
However, only map functions are cached: reduces/shows/lists/updates and others 
are recompiling for each call, so all this import optimization tricks are not 
so useful as they have to be.
I suppose that also would be better to extend preimported packages with most 
popular and useful, which probably would be always imported, such as:
time, datetime, re, hashlib, math, random, itertools and others. But that would 
be very implicitly feature without reading of docs and not only I should decide 
what will be in this list.

No side effect is requirement for views only, afaik, because index is based on 
view result only, while shows/lists just the way to show data in nicer form. 
There is one more thing to keep views as much stable and independent from side 
effects as possible: if you have dozen millions documents last thing that you 
would like to do is to rebuild view index, because this would take hours. Yes, 
trick with secondary server and replacement view index is nice idea, but you 
still have to lose your hours and you'll have service down for a some time.
However, in 1.1.x branch was added feature to require view/lib stored module 
for map functions.

Suddenly for couchdb view servers, hoster wouldn't provide some space for your 
own modules because that would require some additional interface, monitor to 
reload module set in realtime without forced restart of view server and...this 
solution killing portable pythonic couchapps. Javascript couchapps are awesome 
because you just have to type: "couchapp push" and that's all - it works!

So, what resolution will be?

Original comment by kxepal on 24 Mar 2011 at 7:40

[GoogleCodeExporter]

Can you link some documentation on that 1.1.x feature? That's something I'd be 
*very* interested in learning about.

Show and list functions are supposed to be side effect free too. That way, they 
can be cached by couch (though I'm unsure if couch itself actually does that). 
I'm pretty sure couch does proper Etag handling of the show/list results, so if 
you're expecting that you can generate a new result each time someone accesses 
a doc via show, or a set of documents via list, know that couch will *tell* the 
client/browser to use what it already has if none of the pertinent documents in 
the database have changed.

Check out:
* http://guide.couchdb.org/draft/show.html#constraints
* http://guide.couchdb.org/draft/transforming.html#example (see the first 
"lightbulb" blockquote in that section).

Hah, you're right about the map caching. I forgot that some of the others don't 
cache! Hmmmmm. We could do our *own* caching. I'm not sure if that's considered 
bad behavior or not, but I don't see how it makes a difference, and really, I 
think the fact that they send the reduce function *every time* a reduce 
computation is needed is a bad choice in protocol design -- it's simpler, yes, 
but they could have just added 'load' and 'unload' commands for functions, so 
that you can do one-time compilation.

What we can do is cache the reduce/list/show functions they give us and run the 
computation. Next time they pass us a function, we do a string compare on the 
new code for that function to the string of code we originally received for 
that named list/show/reduce function. If it's the same as before, then our 
compilation step becomes a no-op, and we just use what we already had. If the 
function is different, then we assume that the design doc has been updated, and 
we recompile. This way, we can do things like use closures for those 
performance gains. Couch's own rules and reasons for side-effect free functions 
are what gives us the right to do this kind of caching.

Moving on... well random probably shouldn't be imported (or at least not used 
by any couch stuff), since by its very nature, it'll produce different results 
every time.

As for downtime, in many cases, couch can service requests while an index is 
being rebuilt. Also, you can easily replicate to another 
database/server/whatever (secondary server), rebuild it there, and temporarily 
make that the primary database your serving from while you rebuild the index on 
your real primary. That sounds complicated, but as we all know, in couch that 
takes less thought than it took me to write about it just now. Also, that's all 
assuming that couch's index hot-rebuild doesn't cover your use case. 
Hot-swapping couch databases and even couch server instances, or adding 
redundancies and failovers is a fact of life with couch -- sure, there are 
plenty of us running just one couch instance for a given application, but it's 
so painless to temporarily add another copy ad-hoc, and tear it back down when 
you don't need to again. Unlike with other systems, you don't even really need 
to plan ahead when you do it.

You've got a good point. I'm not sure what the resolution would be. Clearly 
python wins over javascript for couchdb not because of its pretty and concise 
syntax, since view/list/show functions are about the same size in either 
language if you aren't allowed to import anything. Python would win out because 
of its standard library (which is API-stable enough for couch), and because of 
its 3rd party modules.

In any case, behavior-changing module upgrades could only be handled by 
rebuilding the index. Even though the code that couch can see (the code stored 
in the design doc) hasn't changed, the code it links to has. So you simply have 
to treat it in same way as if you changed a line of code in the map func 
itself, and there's no way around that.

Just like with retooling your own view/list/show function code, you have to 
strike a balance between the time you need to spend rebuilding an index, and 
the benefits you get from changing the code. After all, you can always choose 
to *not* upgrade your python or module to a new version, and just because there 
is a new version, doesn't mean you need it.

Original comment by extempor...@gmail.com on 26 Mar 2011 at 6:04

[GoogleCodeExporter]

The only documentation I saw is the source code(: 
https://github.com/apache/couchdb/commit/7665e449cdfff1e660ed2bbac3de4507cb063a1
8#share/server/state.js
AFAIK, this command passed automatically if ddoc has views/lib/... path set, 
but I'm not sure. However, I could think in another way while looking on test 
case.

Caching shows/lists/other ddoc subcommands may be possible, but this cache 
would be reseted on each design document update. Reduce functions couldn't be 
cached without source code comparing. However, this trick wouldn't work with 
0.10.0.
There is command ["reset"] to clean up map functions cache and drop all you 
configuration: mime types, reduce_limit etc. However, again, it's system wide, 
not available from the outside.
I need some time for experiments to understand all profits and all flaws for 
such caching. If it hadn't been implemented for javascript view server, there 
must be some reason, right? First one I see, if you update 3d party package 
within system, your cached byte code wouldn't be updated too - design have not 
been changed! - and you'll have a lot of fun in this case(: It could be 
recompiled once again for such fail, but tests are still needed.

I have also found case that breaks idea with imports on top of function:
>>> import datetime
>>> from itertools import groupby
>>> def test(doc):
>>>  yield doc['_id'], 'passed'
the result namespace would be always:
{'datetime': <module 'datetime' from 
'/usr/lib/python2.4/lib-dynload/datetime.so'>,
 'groupby': <type 'itertools.groupby'>,
 'test': <function test at 0x7f3395044938>}
So, those function that iterator would found will be groupby. That's wrong one, 
but it returns two value tuple, but will generate very strange error:
>>> TypeError: <generator object at 0x7fcebfa58908> is not JSON serializable
Totaly crushing view server. You'll have to spend a lot of time with --debug 
option enabled to understand why, but currently it would not help you in such 
case without additional logging. And if generators was JSON serializable you've 
got even wrong result without any warnings. Still not very explicitly and 
relaxing behavior ):
Binding by names? Not an option.

Random module shouldn't be used for views for sure, but it could be useful for 
lists to randomization output.

Idea with swapping temp/production databases is nice too if temporary couch 
instance could serve for a while as production one...but I suppose this 
interesting disquisition not for this issue(;

In next things I'll agree with you - we have to find mostly ideal point of 
balance. Hard optimizations and tricks is part of highload environment. There 
could also be used pypy instead, other faster json module etc. Our task is to 
create tools that are works, works good, but also have some space for heavy 
optimization with some trade off.

Original comment by kxepal on 26 Mar 2011 at 8:09

[GoogleCodeExporter]

Yeah, I'll have to look into that 'require' thing. On first glance, it looks 
like couchjs is doing a request to the design doc for the dependencies.

Right, as said in a previous post, in order for the above-the-function option 
to work, without the use of a closure, you (the view server function compiler) 
would have to check every key in the locals dictionary that exec generates to 
see if it has a __module__ attribute, and if that attribute has the value of 
None. The only backwards compatible requirement we need is that there is only 
one callable object (usually a function) that has __module__ set to None (since 
non-imported local functions/classes will have __module__ of None).

>>> code = """
... from datetime import datetime
... 
... class OldStyleClass:
...     pass
... 
... class NewStyleClass(object):
...     pass
... 
... y = 17
... 
... def test():
...     return 5
... """
>>> locals = {}
>>> exec code in {}, locals
>>> locals
{'y': 17, 'test': <function test at 0x7f7e25f10320>, 'NewStyleClass': <class 
'NewStyleClass'>, 'OldStyleClass': <class __builtin__.OldStyleClass at 
0x7f7e25f196b0>, 'datetime': <type 'datetime.datetime'>}
>>> for key in locals:
...   if callable(locals[key]):
...     if locals[key].__module__:
...       print key, "is *not* a candidate, since it's imported from", 
locals[key].__module__
...     else:
...       print key, "is a candidate (hopefully the only one, or we'll have to 
error out)"
...   else:
...       print key, "isn't even callable, so we don't care about it"
... 
y isn't even callable, so we don't care about it
test is a candidate (hopefully the only one, or we'll have to error out)
NewStyleClass is *not* a candidate, since it's imported from __builtin__
OldStyleClass is *not* a candidate, since it's imported from __builtin__
datetime is *not* a candidate, since it's imported from datetime

Huh, so apparently class definitions inside of an exec will be associated with 
the __builtin__ module, so we'd have to check for that, as well. But in 
general, it's easy to do a backwards-compatible check for non-imported 
callables.

Oh, perhaps the answer to the module distribution problem is to put a custom 
import mechanism that checks for those modules as attachments to the 
view/list/show functions design doc before checking the normal on-disk module 
path. Couch's _changes API would need to be monitored for design doc changes by 
couchpy too, so that couchpy can know when it needs to reload modules. If this 
were achievable, you could bundle your modules in the design doc itself 
(regular zip files and eggs could be supported).

The best way to get a good system in place for this is not to work around 
Couch's API, but instead to work directly with the Apache Couch community to 
support everything we're talking about, since none of it violates the 
side-effect-free requirements of couch if dependency checking can be moved into 
couch. This wouldn't mean that couch would have to understand any programming 
language, but would be able to handle changes to certain special design doc 
keys. For example, couch could *hypothetically* do:

{
  "_id": "_design/app",
  "lib": {
    "calc": "def something_statistical(a,b,c,d): return (a,b,c,d)",
    "image": "#some-package v1.3.2",
    "chart": "#other-package v4.1.7",
  },
  "depends": {
    "views.test": ["lib/calc"],
    "shows.graph": ["lib/chart"],
    "lib.chart": ["lib/image", "_attachments/something_local.egg"]
  },
  "views": {
    "test": {
      "map": "def fun(doc): yield doc['_id'], calc.something_statistical(*[doc.get(k) for k in 'abcd'])"
    }
  }
}

Once again, this doesn't exist in couch, but if it were implemented, couch 
would only need to know how to interpret the "depends" key. If a string in 
"lib" changes (couch doesn't need to know or care what the contents of that 
string mean), then everything that depends on it needs to get updated, just 
like it reindexes views when the view function strings are changed. In the case 
of list or shows, this would mean setting a new Etag that invalidates 
client-cached versions of the previous show/list results. Couch would also need 
to send the dependency to the view server when it's needed, in the form of some 
kind of addlib command. couchpy itself could ignore the # version stubs, since 
those would just be there to provide an easy upgrade path for libraries. Or it 
could compare the version shown there to the version of the module it imports, 
and update the design doc if a new version is found on the module path. 
Dependencies starting with "_attachments" could be handled specially by couch.

Original comment by extempor...@gmail.com on 26 Mar 2011 at 11:26

[GoogleCodeExporter]

By the way, you're right that use of random is side-effect-free. Just keep in 
mind that couch's Etag/caching semantics will make it so that, if an HTTP 
client does proper caching, it'll do a conditional GET request for the 
show/list the next time you ask for it, and unless one of the documents the 
list/show depends on has changed, couch will tell that client that the resource 
has not been updated.

Therefore, your random lists will only look random once per depended-upon 
document update. This is on a client-by-client basis, though. If you have your 
own caching proxy in the middle, or something like couchbase starts having its 
own response cache, then everybody will see the same random results on each 
request, until the next time one of the pertinent documents is updated. This is 
another "good thing" that couch provides, because even though it might hurt you 
5% of the time, it really helps with scalability and responsiveness 95% of the 
time.

Original comment by extempor...@gmail.com on 26 Mar 2011 at 11:33

[GoogleCodeExporter]

> Yeah, I'll have to look into that 'require' thing. On first glance, it 
> looks like couchjs is doing a request to the design doc for the dependencies.
It doesn't but it have access to it via closure. It just have passed to compile 
function as second argument.

Your example could pass and work as "expected", but it just a case. There are
others that wouldn't worked as "expected". There is needed just stable entranse 
point.
May be some kind of decorator would be solution as:
>>> import datetime
>>> def helper(item):
>>>     ...
>>> @main
>>> def mapfun(doc):
>>>     ...
But would it be good, explicitly and clean? Looks like the same as predefined 
function with special name. I suppose there is no so much need in complex code 
block. One node - one function. Libs will take others with exported statements 
as they have been designed to do + eggs as libs to store more complex packages.

> Couch's _changes API would need to be monitored for design doc changes by 
> couchpy too, so that couchpy can know when it needs to reload modules. 
> If this were achievable, you could bundle your modules in the design doc 
> itself (regular zip files and eggs could be supported).
It doesn't needs as if design document have been changed there would be passed 
command to refresh it within local cache.

Also note, that attachments is separate entity that just binded to document, 
but doesn't pass with it. So to call attachment from show/list you have to make 
pure http request - madness!(:

> For example, couch could *hypothetically* do: ...
Too complex solution: instead of just create function you have to create it + 
set up all required dependences to make to work correct. Same thing does 
require function currently - just invoke it and extract needed exported 
statement.

> By the way, you're right that use of random is side-effect-free. Just keep 
> in mind that couch's Etag/caching semantics will make it so that, if an HTTP
> client does proper caching, it'll do a conditional GET request for the 
> show/list the next time you ask for it, and unless one of the documents the 
> list/show depends on has changed, couch will tell that client that the 
> resource has not been updated.
In show/list function you could set your own headers and disable caching via 
Expires header. It has higher priority than Etag one. Actually, Etag only 
__may__ be used for cache proposes.

Original comment by kxepal on 26 Mar 2011 at 1:08

[GoogleCodeExporter]

Good idea! As you indicated, if you have a single callable, then it'll work as 
expected. If you have more than one callable, you must designate it with @main.

Well the main point is, that your idea provides a mechanism for the programmer 
to be as expressive an succinct as they need.

...
def helper(item):
...
@main
def mapfun(doc):
...

is the equivalent of:

def mapfun():
   ...
   def helper(item):
   ...
   def mapfun(doc):
   ...
   return mapfun
mapfun = mapfun()

Only difference is that the decorator is a *lot* easier.

The point about my dependency solution is that it lets couch handle recursive 
dependencies with respect to index rebuilding and Etag handling, so that couch 
can make sure that all data is consistent. I agree, it's complex, and there's 
bound to be a better way (I don't care for my solution either -- it's an 
initial suggestion). I just know that if we have to manage recursive (or even 
non-recursive) dependencies ourselves, then it won't work. Sooner or later, 
we'd end up with a badly inconsistent database, with bugs that are really hard 
to notice.

Dependencies are necessary because in the typical couch application (at least 
all of the ones I've done), there is a lot of duplicate code, and that 
duplicate code makes the application much much harder to maintain.

Do you really want to override Etag handling as done by couch. Put it this way, 
Etag is the absolute best caching mechanism available to you, but it's also 
*very* complex to get it right. Enterprise-grade server software often fails to 
handle it usefully (Apache uses inode numbers, which does not allow you to 
cluster while still keeping out-of-the-box caching), and many high-end websites 
with big budgets never manage to implement it, instead using expires headers, 
or spending money on a secondary server to deal with application inefficiencies.

Couch manages Etags perfectly, so even though it's easy to add nodes to scale 
couch, Etags work in the opposite direction, making it so you have much less of 
a need to scale. If you have something that's completely dynamic, and there's 
nothing in couch's architecture that tells you that you must use idempotent 
show/list functions, then by all means, send a no-cache header. But if you have 
something that is barely dynamic (like you include a random hash with the 
output, just for the heck of it, or you want to add a string saying 'response 
generated in 0.0013 seconds'), then you probably want to rethink what you're 
trying to do, since you're sacrificing a lot to gain so little.

Original comment by extempor...@gmail.com on 26 Mar 2011 at 2:08

[GoogleCodeExporter]

> Only difference is that the decorator is a *lot* easier.
Easier? May be. Implicitly? For sure. You have always keep in mind this @main 
decorator. However, I see we've come to current, original state - single 
function which creates inner context (;

> Do you really want to override Etag handling as done by couch. 
I don't mean to override it, but I've answer to you how to workaround cache 
case based on Etag.
About Etag: 
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.19
a little below you may found Expires header.

Original comment by kxepal on 26 Mar 2011 at 2:27

[GoogleCodeExporter]

At first, sorry for those "wall of text" that subscribers had received from 
us - probably we have to create separate topic on groups, but now it's too 
late. And special sorry to those one, who unstarred this issue - he wouldn't 
receive notification about new version of view server that I would like to 
attach for testing.

Short changelog:

! remove dependency from versioning decorator

! fix Mime class and show functions with provides/response_with methods - they 
was just totaly broken.

! fix Python exception encoding for CouchDB versions < 0.11.0

! correct filters for versions >= 0.11.1. There is no more userctx argument, 
beware!

+ add missed ddoc cache (thanks for this discussion)

+ add support for add_lib command. CouchDB version >= 1.1.x required

+ add support for views command. Currently available for trunk version

+ add support for secobj for validate_doc_update commands. Requires CouchDB >= 
0.11.1. However, this argument leaved as optional due to it doesn't mentioned 
in most examples.

+ add require function with same behavior as same javascript function has

+ add docstrings for most valuable methods with descriptions and examples

~ add support for 0.8.0 version - that was too easy (:
~ allow imports in design functions (see notes below)

~ _log function has been replaced by logging handler

~ correct error message for design function wrong definition

~ code cleanup, reorganisation, formatting fixes

~ more tests added and passed (47 total, 5 failed for 0.9.0 due to I couldn't 
reproduce valid behavior - have someone windows binaries of 0.9.x?)

Something about imports:

http://mail.python.org/pipermail/python-list/2007-September/507450.html

I really hadn't knew about this behavior(: So any imports at top level are 
useless if only they are not be explicitly passed to target function as 
arguments or through decorator. However, I've allow usage of them due to 
perfomance reasons.

More detaled history of changes avaiable in viewserver branch:
https://code.google.com/r/kxepal-couchdb-python-featured/source/list?r=viewserve
r

Next questions that I have:
1. Should I split view.py into view package(propbably better name it viewserver 
package) due to a code growing and missing support of sphinx autodocumentation?
2. Should I add preimported modules? I've stoped at next ones: base64, 
calendar, datetime, math, random, re, time - they are quite common, useful and 
avaiable in all supported versions.
3. Should I add eggs support via --egg-cache parameter where storage folder 
would be specified? Eggs could be stored as base64 encoded strings, not as 
attachments due to they are not avaiable from view server.

Original comment by kxepal on 2 Apr 2011 at 6:00

Attachments:

• view.py
• testutil.py
• view.py

[GoogleCodeExporter]

Ok, I'll answer on those questions by myself(:

> Should I split view.py into view package(propbably better name it viewserver 
package) due to a code growing and missing support of sphinx autodocumentation?
Yes, I should. Because operate with 2K of very nested codebase with massive 
cross functions dependencies is not easy and missing sphinx autodoc feature 
makes to be sad.

> Should I add preimported modules? I've stoped at next ones: base64, calendar, 
datetime, math, random, re, time - they are quite common, useful and available 
in all supported versions.
No, I shouldn't. Because I couldn't decide the developer needs for current 
project, even if those modules are all fits to most tasks. Instead of that, 
I've create something like QueryServer constructor, which could be used to
create your own QueryServer with your own behavior without couchdb-python code 
patching. Petty nice solution, right?(; See `construct_server` function in 
`couchdb.server.__init__.py` for how the default query server is defined.

> Should I add eggs support via --egg-cache parameter where storage folder 
would be specified? Eggs could be stored as base64 encoded strings, not as 
attachments due to they are not available from view server.
Yes, I should. Because this feature provides too much to leave it ignored. 
However, it's optional and must be enabled explicitly for security and 
compatibility reasons. To store eggs within design documents you should encode 
egg as base64 string. See documentation for examples.

So, query server was totally refactored from single module to full package and 
here is new version changes:
+ add support eggs as modules.
+ add option to control GET request to update functions.
+ add query server constructor: define your own context, error handlers, 
commands(if you've own CouchDB fork or living with very nightly builds) and 
more.
+ add query server documentation article.
+ add own logging channel for each part of query server.
~ update "Writing views in Python" documentation article.
~ fix doc strings to make them more sphinx friendly.
~ fix for require circular references (COUCHDB-1075)
- remove debug decorator, because now you may implement it by your own if you'd 
like.

Tested on Python 2.4-2.7 and PyPy 1.5. All changes are still available at 
http://code.google.com/r/kxepal-couchdb-python-featured/source/list?r=viewserver

And that's all I think(: Could someone review documentation articles due to my 
poor english knowledge and code to decide is there something needed to change? 
Any ideas? Criticism? Thanks(:

Original comment by kxepal on 9 May 2011 at 4:25

Attachments:

• query-server.tar.gz

[GoogleCodeExporter]

tested on android 2.3.4 Google Nexus One using Py4A application. To share my 
happiness do next things:
1. copy couchdb package folder to 
/sdcard/com.googlecode.pythonforandroid/extras/python (query server imports are 
absolute and uses couchdb package as root)
2. create file on sdcard, for example /sdcard/couchpy, and place next code into 
it:
PYTHONPATH=/data/data/com.googlecode.pythonforandroid/files/python/lib/python2.6
/lib-dynload
PYTHONPATH=${PYTHONPATH}:/mnt/sdcard/com.googlecode.pythonforandroid/extras/pyth
on
export PYTHONPATH
export PYTHONHOME=/data/data/com.googlecode.pythonforandroid/files/python
export 
LD_LIBRARY_PATH=/data/data/com.googlecode.pythonforandroid/files/python/lib
/data/data/com.googlecode.pythonforandroid/files/python/bin/python 
/mnt/sdcard/com.googlecode.pythonforandroid/extras/python/couchdb/view.py 
--couchdb-version=1.0.0
3. add next line to query_servers section in CouchDB configuration:
python = sh -e /sdcard/couchpy
4. ...
5. now you could use pythonic design documents on android(:

Original comment by kxepal on 22 May 2011 at 4:38

[GoogleCodeExporter]

It's good thing to review your own code after some time passed. This update 
includes a lot of fixes and even some new features:

global:
- removed global state and cross module references (WOO-HOO!)
- rewritten QueryServer api
- added SimpleQueryServer as high level abstraction on top of QS internals
- added MockQueryServer to help write unittests
- fix docstring and typos.
- query server logs are more useful now in debug mode
- update documentation with android paragraph and how to customize query server
- place TODO references to actual CouchDB issues: COUCHDB-729, COUCHDB-282, 
COUCHDB-1261, COUCHDB-898. I could fix them locally, but this will make more 
differences between original JS server and Python one.
- add more over 170 test cases

compiler:
- fix crush for compilation of source code with windows newlines
- fix double crush if function compilation failed
- fix crush for malformed base64 encoded egg
- fix crush on egg cache usage
- code refactoring

stream:
- abstraction from JSON module exception type on decode/encode operations

render:
- fix COUCHDB-1272
- code refactoring

validate:
- prevent query server crush by validate_doc_update on Python exceptions

views:
- reduce_output_overflow error now will be raised properly
- small refactoring
- document seal now works better with copy.deepcopy()

design functions:
- send(), start(), provides(), register_type() available only for show and list 
functions
- get_row() available only for list functions
- log() function is not proxy of logging.info anymore

All test passed for:
- Python 2.4 to 2.7
- PyPy 1.5 and 1.6 
- Android 2.3.4 with Python for Android version 5 against CouchDB-1.0, 
andorid-0.1 and MobileFuton 1.7

Please, could someone review docstrings and sphinx docs? I'm sure documentation 
text is far from good state /:

Original comment by kxepal on 13 Sep 2011 at 10:44

Attachments:

• queryserver.zip

[GoogleCodeExporter]

I took a look at this, but I'm having some trouble getting the tests running. 
In particular, this bit doesn't seem to work, independent of the view server 
used:

djc@enrai couchdb-python $ python
Python 2.7.2 (default, Oct 24 2011, 10:16:20) 
[GCC 4.5.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> pipe = subprocess.Popen(['/usr/bin/python2.7', 'couchdb/view.py'], 
shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, 
stderr=subprocess.STDOUT)
>>> pipe.stdin.write('["reset"]\n')
>>> pipe.stdout.readline()
^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyboardInterrupt
>>> 

Meanwhile, this works just fine from the command-line:

djc@enrai couchdb-python $ python couchdb/view.py
["reset"]
true

I looked at the queryserver.zip from comment 25 and the files from comment 6. 
It seems to me that the former is too large and complex to take into 
couchdb-python at this time. The stuff from comment 6 is much more simple, but 
I couldn't run the test suite due to the above issue.

Finally, the code in comment 6 does a bunch of stuff to stay compatible with 
all of 0.9, 0.10 and 0.11+. I would propose that any new view server code we 
take for our next release be limited to supporting 0.11+-compatible; that's 
already quite old at this point.

Original comment by djc.ochtman on 12 Feb 2012 at 11:49

[GoogleCodeExporter]

Hi, Dirkjan!

Thanks for first review(: Actually, I never run it as subprocess, but if you 
take a look at couchdb/tests/testutil.py::QueryServer so it could be run as 
subprocess, just remove shell=True from Popen.

There is a huge difference between comment 6 and comment 25. It's not in code 
size, it's in bugs, code complexity, documentation, tests, features, logging 
and how easily you could extend it without getting things broken. Support of 
old CouchDB releases is not a little part of it, just a few functions that 
easily could be removed. For example, I've easily added multiprocessing support 
for qs#25 for map/reduce functions just by decorating server/views.py functions 
without touching source code.

Also, you have to change a look from comment 6 to comment 22. That was the last 
version of all-in-one-file queryserver, but it still buggy by design.

Main goal of qs#25 and all code splitting was to simplify future support, allow 
to extend it easy and help with couchapps unittesting because now you could run 
it not as subprocess. And it had been reached.

You may read changelogs in this thread and in my clone at viewserver branch, 
they are quite full.

I admit, that it's a little big patch for about 250KB of code(removing 
docstrings could reduce it by half I sure), but I'd like to take support of it, 
because I use it for everyday tasks, I knew each line of it, I'd like to help 
couchdb-python project and I do not want to create 
yet-another-python-queryserver-project. Peoples knows about couchdb-python, 
knows about his viewserver and expecting that it's fine. Why not to satisfy 
them?

Original comment by kxepal on 12 Feb 2012 at 12:40

[GoogleCodeExporter]

Updated Python query server in attachments. After almost year usage in 
production there was fixed some small problems:
- Eventually crush on chunk encoding for _list functions.
- View lib cleanup on reset command
- Handle single named MIME type params e.g. application/pdf;base64
- Reduced useless logging output to improve they readability.
- Fix COUCHDB-1330.
- Fix crush on malformed MIME type.
- Couchapp modules no more needed to be wrapped into some scope: just write 
regular 
Python code for them. For example:
{{{
  import datetime

  def foo(datetime=datetime):
      return datetime.datetime.utcnow().replace(microsecond=0).isoformat('T')

  exports['foo'] = foo
}}}
Now you may remove any proxy hacks to have simply and expected code behavior:
{{{
  import datetime

  def foo():
      return datetime.datetime.utcnow().replace(microsecond=0).isoformat('T')

  exports['foo'] = foo
}}}

This change doesn't affects on other ddoc functions: show/lists/views etc.

Original comment by kxepal on 3 Aug 2012 at 4:54

Attachments:

• queryserver.zip

[GoogleCodeExporter]

Original comment by djc.ochtman on 21 Sep 2012 at 8:33

[GoogleCodeExporter]

This issue has been migrated to GitHub. Please continue discussion here:

https://github.com/djc/couchdb-python/issues/146

Original comment by djc.ochtman on 15 Jul 2014 at 7:19