Merge feature count changes

Bookworm-project / BookwormDB

Tools for text tokenization and encoding

MIT License

84 stars 12 forks source link

Merge feature count changes #59

Closed organisciak closed 9 years ago

organisciak commented 9 years ago

Stream input for feature count ingest
Remove seen file loading/saving from feature count ingest

bmschmidt commented 9 years ago

One question about the behavior of the logging module. I'd guess that the calculations in the following line must still happen when the logging level isn't set to debug:

logging.debug("Token batch attached (%d s)" % int(time.time() - start))

The overhead of coercing a time.time() call to int probably isn't enormous, but is it worth wrapping this in if logging.level == "DEBUG" or whatever the equivalent is?