Improve modularity

[swiftarrow]

Note: Move further discussions to separate wiki as appropriate EDIT: I've re-named this issue to reflect better what it really attacks.

PROBLEM: The current situation is that we have multiple apps for each type of record. Each app has it's own delete function, own "entry on the map" function, etc.

SOLUTION?: Basically, our app structure should look more like:

_App: Record Logging_ --presents a single base "mother class" of the records. --Different types of messages should be described in a config file or some easily altered form. Based on this description the record input form should be dynamically generated.

Sample description:

RecordType         13
RecordTypeName              "Road Assessment"
RequiredFields              GPSLocation, Quality
RecommendedFields     Comment

In turn, the type of input of each field should be defined. Such as:

Quality: float, possible values: -1, [0, 1], Input form: slider from 0 to 1, checkbox for unknown (-1)
Comment: freeform text field, Input form: text box.
DeleteRecord: Boolean, default FALSE.  If TRUE, the record is marked as "deleted" and no longer displayed on the map or in lists. (This message is also sent to other devices, who do the same).  Input form: greyed button, red when selected.

This wiki page can be used as a guide for this: https://github.com/WiseEarthTechnology/CrisisCommunicator/wiki/Saving%2C-Storing%2C-Sending-Messages

the wiki entry may need to be edited, if some record types are found incomplete.

In the rendering of the input form, fields must be separated into "required" and "recommended" categories. JQ code should be used to make the form display appropriately.

_App: Map Display_ --provides the main interface to the CrisisCommunicator and defines interaction with different records on the map (delete, etc).

_App: Show Refugee Center (future work)_ --displays details of a certain refugee center - the people, the resources, etc. Does not need to be a Map based interface for this, but we can have things like "plot the locations of where the people in here came from on a map".

Etcetera.

_TODO:_

• [ ] Clearly understand how this should work in code
• [ ] Create Sub-issues to implement this functionality as issues, and comment below with cross-references
• [ ] Solve sub-issues
• [ ] Mark this as complete

[swiftarrow]

I guess step one for this would be to codify the Record Templates in a settings file, and to create a file of psuedocode for the different types of fields.

This I can do. Is there any specific syntax that you seasoned programmers recommend for such a file? Otherwise I'll try to use python's dictionary syntax, like:

recordtemplate_roadassessment:
    nicename = "Road Assessment"
    code = 0x13
    required = [GPS, Quality]
    recommended = [Comment]

Would that work? Is that syntax correct?
Would it be better to call it recordtemplate_13 for the purpose of sorting? Let me know your input then I'll start on that.

[swiftarrow]

Er, I didn't mean to close the issue :)

[swiftarrow]

Can I please have feedback on that last comment?

[varrunr]

That sounds like a good idea, it would make easier to create message formats on the fly. We need a module to understand the message file syntax. While one could use XML, it would be hard for a non-technical person to create new formats. Creating a custom format(like python style) might make it easier to edit/add new messages, but would mean implementing our own "parser" which understands the format - instead of using standard parsers(e.g. XML parsing libraries), which means write a lot more code and will require a lot of error handling. We need to find a tradeoff between the two of them. Let me research and see if there one that fits the description.

[swiftarrow]

DO you think I could just write it in plain Python, as a Dictionary?

Thanks, Love, Peace, Link

On Fri, Nov 8, 2013 at 11:50 PM, Varrun Ramani notifications@github.comwrote:

That sounds like a good idea, it would make easier to create message formats on the fly. We need a module to understand the message file syntax. While one could use XML, it would be hard for a non-technical person to create new formats. Creating a custom format(like python style) might make it easier to edit/add new messages, but would mean implementing our own "parser" which understands the format - instead of using standard parsers(e.g. XML parsing libraries), which means write a lot more code. We need to find a tradeoff between the two of them. Let me research and see if there one that fits the description.

— Reply to this email directly or view it on GitHubhttps://github.com/WiseEarthTechnology/CrisisCommunicator/issues/2#issuecomment-28104714 .

[varrunr]

I just want get a few things clear

1. What is the problem you are trying to solve? Are you trying to make it easier for a user to enter new record types? Or are you trying to make it easier for a developer to add new record types? It is as per my understanding that you want record templates to be dynamic instead of hardcoding them into the application. If the target for this is is the user, it should be simpler than writing python syntax?
2. I have not yet used the webapp, is it related to any functionality present right now, could you give me some context? Because as per my understanding the job of understanding the messages should be better tasked to the message daemon.

[swiftarrow]

You're right, I want to make the record templates dynamic, for these reasons: * Making them dynamic will enable the application to be flexible; adding / changing functionality as per the requirements of (for example) Red Cross. * The display of fields on the application's form can be standardized and automated to show only the fields pertinent to the type of entry. (IE, prominent display of required fields, not-so-prominent display of recommended fields, and hidden other fields). * The code will be more generalized. Right now, for each record type implemented, there is a separate "app" in the django project, whereas it should be a single app for all records. This means that, for example, the "delete record" function is written for each record type, leading to code duplicity, and making it more difficult to maintain.

Per my (not-quite-programmer's) understanding, the message daemon should take the entered fields and formulate them into messages, and vice-versa.

I hope this makes sense!

Thanks, Love, Peace, Link

On Sat, Nov 9, 2013 at 1:33 AM, Varrun Ramani notifications@github.comwrote:

I just want get a few things clear

1. What is the problem you are trying to solve? It is as per my understanding that you want record templates to be dynamic instead of hardcoding them into the application
2. I have not yet used the webapp, is it related to any functionality present right now, could you give me some context? Because as per my understanding the job of understanding the messages should be better tasked to the message daemon.

— Reply to this email directly or view it on GitHubhttps://github.com/WiseEarthTechnology/CrisisCommunicator/issues/2#issuecomment-28109530 .

[swiftarrow]

Hey guys! Greetings from Budapest! Any feedback?

[varrunr]

Give me some time to read the existing code and try to get a cleared understanding of the problem.

[swiftarrow]

I've edited the issue description. I think it makes it a little clearer. I also found this link helpful: http://www.philipotoole.com/reusable-django-applications

[RanjithP]

This is a summary of my email exchanges with Linkesh: -- Regarding the idea of creating a single model for the record types instead of having a separate model for each of them:

This is a good idea but the data design will look slightly more complicated and not very explicit.

Creating a single model with a dynamic way of creating the fields depending on each record type might take away some of the "magic" which django provides. Eg. Automatic form creation from a model might not work..anyway this will need to be tried out for a simple use case first.

-- Regarding the idea of re-designing the database:

Some of these spatial databases are implemented using an object paradigm rather than a relational one. eg. Postgis. And I think there is a reason for it. Its very intuitive to map geographical space objects (Lat, Long) to one or more class objects which in turn have attributes. In this project maybe we could map objects of a class such as HAZMAT, RoadAssessment etc. to a geo space object.

ZODB for python has been around for quite a long time. Its an object database. The Plone Content management system uses it.

One of the obvious advantages would be in not using an ORM (Object relational mapper) which Django uses to map your relational database records in to objects. Another one would be that there is no separate process for your database engine, ZODB runs in the python process itself. These would be obvious memory savers.

One obvious disadvantage in using ZODB would be no automatic form creation from models, since forms will have to be manually created. (This is as per my present knowledge).

http://www.zodb.org/en/latest/ http://www.ibm.com/developerworks/aix/library/au-zodb/ https://pypi.python.org/pypi/django-zodb/0.2rc1 (not yet stable project)

[varrunr]

Hey @RanjithP , ZODB sounds interesting. I do not have experience with spatial databases before, but is worth investigating. Some questions to answer

1. How much memory does ZODB save when compared to SQLlite? Memory profiling will have to be done.
2. How would a single model for all record types look like? How much effort would it save in terms of code written etc?

One obvious effect of creating a single model for record types with dynamic field creation of fields as in an earlier mail would be not so-easy-to-understand database logic.

That should be fine because we really don't have a complex schema at present as well.

One obvious disadvantage in using ZODB would be no automatic form creation from models, since forms will have to be manually created. (This is as per my present knowledge).

Forms are just one of the convenient features of Django, but is not a absolute necessity. We can always write some wrapper classes for easy creation of forms.

In the meantime, let me checkout ZODB.

[swiftarrow]

Some main objectives to satisfy are:

• Though each record has different "recommended" fields and different "required" fields, each record is treated exactly the same in the network and the display. The model would be something like "MapAsset", not "RoadAssessment", "Refugee", etc.
• To dynamically expand the system to include additional record types, by sending a "New Record Template" message specifying the required and recommended fields.
• While each record type has required and recommended fields, this should not be an absolute limit; the operator should be able to add data to any field at his/her discretion. So it makes sense that they are all one single model.

When I say "a single model which morphs", what I mean is that the display side is tailored to show different sets of fields based on what type of record is being entered. The code remains the same.

[swiftarrow]

One remark on the database level stuff... I know that Django abstracts the entire db layer into object models. It's best to stick with this object paradigm, meaning that we don't really need to worry about the DB structure.

RE automatic form creation:

With a single model, perhaps we can have a single form template, that dynamically morphs based on the "required" and "recommended" fieldsets. There might need to be some trickery involved here. I imagine the following rough sequence of events:

1. the "add_record" view calls a form template, passing a dictionary of the required and recommended fields which are displayed nicely for the user.
2. The form template has javascript enabling it to (on demand) dynamically browse a list of available model fields and add them to the form, should the user want to. This list can be built by using python to dynamically generate some javascript function, or (as you said) an xml file for the javascript to parse through.
3. Posting the data back to the view would save the posted fields which were populated by the user, leave the unpopulated views as False (or something), and save the record.

Again, by "morph" I mean, the display changes, hiding things from the user.

[swiftarrow]

From our IRC discussion:

My thinking is (and I'm open to a better way): A single model called "Map_Asset". This model contains ALL the fields that could possibly be used by a record type. When creating a road assessment, we would create an object of Map_Asset. When creating a refugee center, also an object of Map_Asset. etc. The display of these records and the fields that they use would be governed by their record type. They wouldnt necessicarily use all the fields for each record type The main issue is to reasonably satisfy the objectives above.

Perhaps (my imagination running again): fields in https://github.com/WiseEarthTechnology/CrisisCommunicator/wiki/Saving%2C-Storing%2C-Sending-Messages need to be enumerated and described, so that we can decide which data fields are necessary in the model for example, HAZMAT asssessment, and Road Assesment fields could conceivably be the same variable in the model

The message daemon will identify fields and record types using the hex codes while that django code can identify them as named variables somewhere, we need to have something that describes the record types in terms of required, recommended, >and maps the variables to hex codes could that be another app? or some lower-level python code? that description would need to be referenced for the display, enteing (form creation), and message sending/receiving

[swiftarrow]

TODO: Move this discussion into a wiki

[varrunr]

Oops, I don't know how I ended up closing the issue. Lets not change the name of the issue.

[RanjithP]

To @varrunr 's questions, check these links out (slightly old though): http://www.blueskyonmars.com/2005/06/18/zodb-vs-pysqlite-with-sqlobject/ http://pyinsci.blogspot.in/2007/09/zodb-vs-relational-database-simple.html

[varrunr]

@swiftarrow @RanjithP Moving the DB discussion to a new issue #31 . The issue is more of a question of which database to go with and has nothing to do with the "single model" idea, because I believe a single model can be achieved with both, so lets create a separate issue for that if needed, but keep it out of this.

[swiftarrow]

@varrunr @RanjithP So, to bring this discussion back to the core issue: how do we implement this? Varrun had recommended writing a meta app which holds the class, and having sub apps inherit the class from there as an intermediate step. Is anyone free some time this weekend to discuss this and maybe chart a roadmap? Thanks!