Backend tasks and indexing

[remram44]

The 'indexer' app shall provide the indexing and discovery operations, to be started from management commands (either debug tools to perform a specific task, or batch operations started from a cron task).

The indexer does basically three things:

• Scanning: this is the action of finding FTP servers in a network. This is useful for LAN installs where the number of addresses is limited to have fully automatic operations. It might be disabled if we only want to monitor specific Internet sites. This can take time but is not very demanding for the app; it has to be parallelized if we want a reasonable scanning rate.
• Checking status: this has to be done frequently and is relatively inexpensive. We just check the status of all the known servers.
• Indexing: this is the act of walking over all the directories in a FTP server to discover files and update the database. Different FTP servers could be indexed in parallel, although this might not be critical.

In addition to scanning a specific IP range, the visiting user can be scanned for an open FTP server. This is useful if the subnet is much larger than the actual number of connected machines, if we assume that the FTP servers want to be indexed in the first place.

This duplicates #22.

[remram44]

Not far now! We're missing SEARCH_ON_USER and some kind of locking to prevent cron commands to run in parallel.

This needs a lot of real-life testing as all the new features were never tried.

[remram44]

I'm moving the SEARCH_ON_USER thingie to next: issue #29.