Splitting into separate apps

[XaviP]

Collecting info about ways to separate an app into several.

As a first step, models.py and views.py can be splitted into separate files. This can be an improvement to avoid merge problems, if different developers works in the same app. Can be better for document each model and view too. Maybe this can be a good idea for valuenetwork/valueaccounting/models.py (12k lines) and valuenetwork/valueaccounting/views.py (13k lines) as this app is like the core of the project, so no need to separate it in different apps (not sure, please correct me):

• How to split models.py into separate files?
• Breaking Apart Models in Django
• How to Split Views Into Multiple Files

[ghost]

Great to meet you OCP team :)

I agree and actually commented yesterday with @bum2 something along these lines in a Telegram chat. I'll write a brief summary of what I think.

This can be an improvement to avoid merge problems, if different developers works in the same app

100% correct. Today if I want to pull some changes from OCP into DEEP for instance I'd have a big pile of difficult to merge conflicts. Plus not only across forks, also, as you mention, across branches.

Maybe this can be a good idea for valuenetwork/valueaccounting/models.py (12k lines) and valuenetwork/valueaccounting/views.py (13k lines)

I could break down, at least, into two modules: core and manufacture. Also make Sensorica specific stuff (related to manufacturing) into a different app inheriting the base classes. Imho this could be, ideally, the way to follow for the different forks. One core app: valueaccouting; made of different generic REA modules. And then users of that app. Kind of following the Pinax project spirit.

So whichever project interested in valuenetworks can use valueaccounting as a dependency for its models and perhaps some views.

But this is kind of an ultimate goal. A first refactoring step perhaps would be breaking down into modules (users, demand, supply, inventory, etc) the views.py file. And also accommodate to the new Django class based views whenever is possible.

I'd be happy to participate in that work. Perhaps we'd temporarily refork your repository for this task, work on pull requests and then integrate them back into DEEP. I'll ask Numa :)

[bhaugen]

Another way to think about it is to separate the main REA concepts into separate apps:

• Resources
• Events
• Agents and then
• Processes
• Exchanges and maybe
• Recipes and Planning

Might also want to separate out apps for some specialties:

• Digital currency processing, or just Faircoins
• Invoicing a la Freedom Coop

Another possibility, that could go along with separating into apps, would be an overall set of abstract model classes and navigation to tie things together.

[bhaugen]

P.S. I think it would be good to write up something for the Django users group to get some advice and maybe even some help. This could become an important Django project. People talk often about wanting to do ERP-like apps in Django, and if we separate out some reusable apps, they could be used in a lot of settings. I also think that economic networks are emerging all over the place, and NRP-like systems will become more important than individual enterprises as in ERP.

I'll try to write a rough draft later today and pass it around for comment.

[bhaugen]

One more thing: I think we should set up a new repo for the refactoring, or maybe more than one if we end up with more than one approach to refactoring that people want to try.

That way, the different forks could keep going, since production systems might be using them until the refactoring gets stabilized.

The refactoring repo could start with a set of internal refactored apps, that could then be made into their own app repos if and when they look good.

If I remember correctly, this is how Pinax did things at first.

[XaviP]

Hey! I'm a bit confused about terminology. I see clearly how to split the file valuenetwork/valueaccounting/models.py into a unique package called "models" (a subdir with __init__.py) which contains several files (resource.py, events.py, etc..). Same with views.py. This way, a live site won't be affected in the update. But if we talk about split the app valueaccounting into several apps, I'm not so sure that a live site keeps unbroken. I mean, I don't know if the new apps re-do their own tables in the db, with data loss in old tables. I don't really know how django apps behave in a change like that. Do you?

[bhaugen]

But if we talk about split the app valueaccounting into several apps, I'm not so sure that a live site keeps unbroken. I mean, I don't know if the new apps re-do their own tables in the db, with data loss in old tables. I don't really know how django apps behave in a change like that. Do you?

I think so. If we separate the system into different apps, it will break the live sites. That's why I think we need to keep the forks going as long as a live site is depending on one.

The apps will set up their own database tables, prefixed with the app name, like appname_modelname. The columns could be the same. The data would need to be migrated from the old tables to the new tables. Easiest way might be dump into json file, change table names in json file, load into refactored db.

[XaviP]

Easiest way might be dump into json file, change table names in json file, load into refactored db.

Ok, then we'll need a new repo and we'll have absolute freedom for refactoring in differents apps. It sounds good to me.

[fosterlynn]

Should the new repo start with the DEEP version? I believe it is a pretty recent fork of OCP, except @bhaugen has cleaned up so the OCP hard-codings are gone. And maybe @escanda has also done a few things.

Would need to check if OCP has had bug fixes since the fork and merge them in.

[ghost]

Should the new repo start with the DEEP version?

I think it could be nice if I would finish the current issue (user invitation into the system), today or tomorrow tops, and also if @bhaugen would help me on the travisci branch so we have continuous integration in the new repo. Also the travisci repo has a few other enhancements.

Once I am done with user invitations I, or anybody else who wants to, can take on the task of diff'ing OCP & DEEP and merging bug fixes as well :)

[bhaugen]

Rough draft of post to https://groups.google.com/forum/#!forum/django-users

Please suggest, or just make, improvements.

Dear Django people,

We are working on an open-source Django system that has grown both in terms of user and developer communities and also size of code base. It is in production with 370 users that we know of.

It has several names, but one is NRP, which stands for Network Resource Planning system, as contrasted with ERP, which stands for Enterprise Resource Planning. NRP has the same scope as ERP, but is for economic networks, not single enterprises. And it is aimed at communities who are experimenting with cooperative economic systems.

We want to refactor the system. It has become too big and unmanagable. So we are looking for advice on how to refactor.

Here's the original repo: https://github.com/valnet/valuenetwork which is being used by http://www.sensorica.co/ Here are the active forks: https://github.com/FreedomCoop/valuenetwork used by https://freedomcoop.eu/ https://github.com/gopacifia/DEEP planning to be used by http://gopacifia.org/en/

Here's an overview slide deck of the the original system. The forks have added other features. https://speakerdeck.com/mikorizal/1-nrp-overview

We think if we refactor this software, many of the features could be reusable by other projects that are not necessarily aimed at the same kinds of user communities. For example, any supply chain or business joint venture is an economic network, and many LLCs are actually networks under the covers. Moreover, we think that economic networks and ecosystems are the wave of the future, more than individual companies. But some of the features of this software base would also be useable by individual organizations.

The system is based on the Resource-Event-Agent (REA) model, which is a much simpler basis for business systems than the usual ERP model. Here's how REA applies to supply chains, for example.

Here's our current discussion about how to do it: https://github.com/FreedomCoop/valuenetwork/issues/215

Anyway, we will be grateful for any advice, and if anybody wants to help, that would be even better.

[Edit: sent] https://groups.google.com/forum/#!topic/django-users/9OU0TfwcmTQ

[bhaugen]

@escanda

if @bhaugen would help me on the travisci branch

I'm getting to that today. All commits to the master branch pass the tests. Moving to the travisci branch in the next hour or so.

[bhaugen]

Maybe should put an REA overview into the message to django-users, too... [edit: done]

[XaviP]

You can make explicit that this sofware is in production with many users.

[bhaugen]

You can make explicit that this sofware is in production with many users.

Done. Also added links to the user orgs.

[XaviP]

One thought: if the final goal is to make a downloable package from pip or something like that, I think it has no sense in splitting in so many apps, as mostly they depend in the other ones. So when one installs it in django, he/she'll have to setup many apps in settings.py Maybe it's better a simple app very well modularized.

[bhaugen]

One thought: if the final goal is to make a downloable package from pip or something like that, I think it has no sense in splitting in so many apps, as mostly they depend in the other ones. So when one installs it in django, he/she'll have to setup many apps in settings.py Maybe it's better a simple app very well modularized.

Good question. I think some of the separatable apps could be used independently, like Agent and Resource. See:

• https://github.com/holodex
  • http://holodex.enspiral.com/
• https://github.com/valueflows/resourceful

But we might want to have a whole kit and kaboodle pip installable too.

[bhaugen]

One more idea: let's figure out what is the minimum set of changes that would allow everybody to use the same repo for the core of the system, and separate out only the differences for the first stage of refactoring.

I don't know off hand what that would be, and it wouldn't cover the original version that Sensorica is using now, just the OCP and DEEP versions.

If we can get one version that both OCP and DEEP could use, with separate apps for their differences, then we could upgrade the core repo to the latest version of Django before we do more refactoring.

[fosterlynn]

One more idea: let's figure out what is the minimum set of changes that would allow everybody to use the same repo for the core of the system, and separate out only the differences for the first stage of refactoring.

:+1:

Probably best to look at all the commits, but off the top of my head, here is one option: starting with the DEEP version, making the FairCoin interface configurable, include @bum2 's changes to OCP after the DEEP fork, and make the OCP hard-codings configurable (if needed, I'm not sure how @bhaugen handled that in the DEEP version).

[bum2]

If you think is good to use the General app, which is proposed in the 'project-planning' branch, then that can be a basic module for mapping resource types, exchange types and skills (like it is now in test at testocp), but also entities, locations, resources (artworks) and the most important: Relations. The general app approach has unused but ready the whole set of triplets 'object-relation-object' between entities, and between entities and arts, and resources, spaces, etc, etc. The tree of Relations can be used and configured to our different needs, etc. The modularity from that point of view is more about adding logic or functionality, like the Welcome app we did for membership process (2014) ... now we have incorporated in OCP a meta 'join-a-project' system with custom forms... modules can be faced also like:

• SimpleTasks
• ProjectProfile (Inventory?, Relations?)
• JoinRequests (memebrship process+welcome) with fobi forms,
• Communication (comments threads at different levels -> future social network?),
• FairCoin Wallet
• Exchanges (multicurrency)
• Orders: Planned Work + Planned Tasks, Recipes, Process ...

But i agree with @XaviP that split the code in files and not the app in apps is easier... nevertheless it depends on where we want to go. If we want to make full use of the General structures, then we can let in the Valuenetwork and Work apps only the logics, the views, the features... I don't know, just first thougths, still thinking...

[bhaugen]

I think we need to work out a principled relationship between your General ontology and the REA ontology. Not good to have dueling ontologies.

[numapanuma]

If we can get one version that both OCP and DEEP could use, with separate apps for their differences, then we could upgrade the core repo to the latest version of Django before we do more refactoring.

In the FC tech chat, Enric mentioned that he would like to see all DEEP planned features included in the main OCP branch. In this case we would just need to figure out what DEEP would use from OCP.

But in more general terms I think it will depend on each user/agent to choose what to use or not. Also because a new user will need to "learn" the platform and if all tools are available at once it might be too overwhelming. Ideally the admin team installing the platform would untick all tools that are not needed to them. Much like Odoo/Wezer/MAP right now. And every agent should be able to enable or disable those tools too, IMHO.

[bhaugen]

A bit more thinking about the General app:

It is already separated out into its own app, so depending on how entangled it is in the logic of the rest of the system, other groups could decide to use it or not.

That will also be the case with several other features that arose from one group and the other groups will need to decide if they want them or not. That includes the planned DEEP features and maybe the whole work app.

Another one is the faceted classification, which also overlaps with the General app. I don't personally care whether the facets or the General survive the merger, but I think it would be a fatal error to damage any of the REA concepts, relationships, or logic. If you don't know why, I can explain at sleep-inducing length.

[bhaugen]

@numapanuma wrote:

in more general terms I think it will depend on each user/agent to choose what to use or not. Also because a new user will need to "learn" the platform and if all tools are available at once it might be too overwhelming. Ideally the admin team installing the platform would untick all tools that are not needed to them. Much like Odoo/Wezer/MAP right now. And every agent should be able to enable or disable those tools too, IMHO.

I think that would require separating into apps, in the Django architecture: as in the installed_apps list in settings.py.

[fosterlynn]

@bum2 do you think we need to merge in the project-planning branch before doing the upgrade? I think it might make it easier if we do. Then let's think about those implications. If not, then I'd suggest we wait to merge, and you can keep working in the branch until it is done.

That decision will help set the course in some ways.

[fosterlynn]

In terms of longer range thoughts about the general app, I think there will need to be some different offerings based on how the larger scope networks (like Freedom Coop and GoPacifia) want to operate, in terms of degrees of autonomy vs centralization. I think different networks will have different thoughts, especially as they are all experimenting.

Besides the question of tree based taxonomy vs facet values, @bum2 's integration of his general app (as far as I understand the implications) tends towards more centralization of the configuration, that is every group will use the agreed upon taxonomy of resource types and exchange types. It envisions a more coordinated network, and emphasizes inter-trading more than internal operations of each group node (collective, project, etc.). Alternatively, one could tend more towards allowing groups to set their own configuration tailored exactly to their needs, without having to agree with everyone else, and see everyone else's configuration all the time. Or one could create some combination so that inter-trading and supply chains are encouraged within the larger network but individual groups can still configure their own operations specifically and privately without that data bothering others and vice versa. Bumbum and I have been back and forth a lot on this in the Freedom Coop context, and have not yet managed to discuss it with admins or project coordinators besides Zumbrunn, mostly due to people being too busy.

Bumbum, I'm trying to be very fair here, so please correct anything I said that doesn't reflect your thoughts. I am fine with whatever Freedom Coop decides (or implements without deciding for that matter). But I do think that if the general app is used, it should be done in a way that other networks can plug something different in. Ah the wonderful possibilities of this modular refactoring!

It may be that the current work app should become the general app integrated app, at least for the purposes of upgrading the Django versions (assuming @escanda hasn't done too much work there).

[numapanuma]

It envisions a more coordinated network, and emphasizes inter-trading more than internal operations of each group node (collective, project, etc.).

We would go for that too. But if:

one could create some combination so that inter-trading and supply chains are encouraged within the larger network but individual groups can still configure their own operations specifically and privately without that data bothering others and vice versa.

That would be awesome.

[fosterlynn]

one could create some combination so that inter-trading and supply chains are encouraged within the larger network but individual groups can still configure their own operations specifically and privately without that data bothering others and vice versa.

That would be awesome.

Yup, I agree, that would be pretty nice all around.

[bum2]

All the REA concepts, relations and logics are great and are the ones the valuenetwork provides. They can be mapped into General but only if we wish (resources as artworks, agents as humans, locations as spaces, etc), but the point is more how far we want to go inside the valuenetwork app or into the work app... what about making a relations of 'needed' or 'wished' resource types?, or skills 'learning', 'teaching', 'searching' ... i mean, the multiple relations and possible views and uses of the system... Also the units and unit types can be treated as resources and resource types, even in the wallet, and achieve a multi-unit accounting and multi-totals counter (timebank, mutual credit, faircoin, cryptos, fiat currencies).

Basically the models are Data in the db, so i think more about common data related to common Apps, and particularities of each 'fork' related to additional custom Apps build on top of that general-common tables of data (which we can share, like the born of a common ontology of types, skills, relations, a repo of worldwide entities, locations, etc)... just the info that is intended to be 'public' or common... Then the custom apps of each network can hold the private information of memberships process, tools like invoicing, the custom types required, etc...

The 'project-planning' branch already uses types and skills from the General app, and filters them by context, so types and skills can be defined 'only for a project' or choose a broader context like 'the whole sector' or 'the whole ocp'. That already works in test. What is missing, to include in a future release, is the abbility to use the types more like Tags, so multi-choices to define resources or exchanges, and from each 'tag' we inherit behaviours or filtered dropdowns, etc... That will fit better with the faceted system, but we'll be producing some more facets and lots of facetvalues and will need the trees to manage them (not plain lists), because we'll need a facet for every big branch and a facetvalue for every branch or sub-branch, and apply them more as multiple Tags to define something. Each facet or facetvalue is one2one with a artwork-resource type branch in the General trees, so if many 'types' are added to define a resource (tags), then many facetvalues are related the resource. Same with exchanges and exchange types. A field that is already in each General type is called 'clas' and works for many purposes, one is to define the name of a django class related the type, so if an object has the type it can inherit that class definitions or methods... I actually use this 'clas' field mainly to have identifiers in order to filter branches easily when needed, to record css classes or whatever.

[bhaugen]

I think those decisions should be driven by real-life use cases from one or more of the groups.

[bhaugen]

Another set of decisions will be about how to bring in new users. OCP has a set of protocols for onboarding new users. It would be good to document them (more clearly than now) and then see if either GoPacifia or Matrioshka wants to use them in every detail, or with some variations. Or if not, what do they want to do about onboarding new users. That's been pretty random in NRP, just done on request by some superuser. OCP introduced the first protocols for becoming members of Freedom Coop and also for becoming members of projects within OCP.

[bhaugen]

Also the units and unit types can be treated as resources and resource types, even in the wallet, and achieve a multi-unit accounting and multi-totals counter (timebank, mutual credit, faircoin, cryptos, fiat currencies).

Currencies should be treated as resources and resource types, but not all units. Like, a liter or a meter are not resources, they are just units of measure. Currency units of measure and currency resources are a little odd in that the same name applies to both, and it seems like a duplication. Probably is, but I don't know how to eliminate the duplication. However, currencies at rest are stored in something, and whatever they are stored in (piggy bank, bank account, FairCoin Wallet or Address) is a resource.

(Did that all make sense?)

[fosterlynn]

Currency units of measure and currency resources are a little odd in that the same name applies to both, and it seems like a duplication. Probably is, but I don't know how to eliminate the duplication. However, currencies at rest are stored in something, and whatever they are stored in (piggy bank, bank account, FairCoin Wallet or Address) is a resource.

I think what you said is exactly how to eliminate the duplication. The resources are always the stock or storage. "FairCoin", "Canadian Dollars", etc. would never be a resource or resource type, only a unit of measure. (This is my conclusion from Sensorica's experience.)

[bhaugen]

Ok with me.

[fosterlynn]

The 'project-planning' branch already uses types and skills from the General app, and filters them by context, so types and skills can be defined 'only for a project'

@bum2 is this a recent change? Last I tested exchange types in testocp, they could not be defined only for a project. But if so, awesome!

[bum2]

About the membership process, i have placed the flexible multi-project system with custom forms and so on but the initial FC membership workflow is not using that system... In order to merge this (will be wonderful for modularity) what is required is to separate the 'share' payment and wallet behavior from the join-a-project common stuff, like the management of the join-requests from the project view, communicating with and accepting or declining memberships, etc. The FC members project can have a Tag or special Type that makes it behave as requiring a share payed with faircoins, after the form is filled and the user is created as candidate. That part can happen or not in other projects, but the whole join-a-project flow system can be totally common.

About the Units as Resources, i think they are defined by their type, and in some cases a resource can act as a Unit, and we can manage multiple unit accounting, by considering different currencies or hours in a timebank (as 'value units'), but have othe totals for liters, kilos, etc. From the General view, units are non-material artworks (creations) like 'resources' and have types to define them and records to define their ratio relations.

@bum2 is this a recent change? Last I tested exchange types in testocp, they could not be defined only for a project.

What i've added is the abbility to 'edit' an exchange type by his creator (or a superuser) and there he can choose a different context for the exchange type recently created... like in the 'new resource type' or 'new skill' (works similar). This is not yet including the edition of the transfer-types sequence, but we can do it later (even editing each transfer in the exchange type). Actually the sequence is inherited from parents exchange type, and the resource branch for the 'main' transfer (inherit_types=True) is defined for this 'custom' exchange type, so when in the exchange logging page the dropdowns are small.

I've placed two docs about the situation in the 'project-planning' branch https://github.com/FreedomCoop/valuenetwork/files/797991/OCP.General_1.docx and about the General app itself https://github.com/FreedomCoop/valuenetwork/files/797990/GeneralApp_1.docx (sorry, github didn't get my .odt nor my .pdf, so i converted to docx and that worked)...

[bhaugen]

in some cases a resource can act as a Unit, and we can manage multiple unit accounting, by considering different currencies or hours in a timebank (as 'value units'), but have othe totals for liters, kilos, etc.

I doubt that you can make liters and kilos into resources in any meaningful way. Each of the REA concepts has a reason for existence, and shit won't work if you misunderstand them.

[bum2]

I doubt that you can make liters and kilos into resources in any meaningful way. Each of the REA concepts has a reason for existence, and shit won't work if you misunderstand them.

Hey man, I agree totally with the REA concepts, but as you said they are both sometimes (resource types and unit types) and for me is not a problem, coexists even as a concept with an history, or other 'classes', being the same name, from different perspectives...

[bum2]

in a exchange office, the resources we manage are currencies, even physical fiat and digital fiat, social currencies, etc... how can we track them in ocp? as resource types, or?

[bhaugen]

Resources and resource types and units and unit types are not the same, even if they have the same name. They are functionally and conceptually different, and behave differently economically and in accounting.

Currencies seems to be confusing, but if you are tracking quantities of currencies, like how many FairCoins do you have, they are resources.

[bhaugen]

P.S. if you want to think about liters as a resource, you need to specify, liters of what resource type?

Liters of milk? Liters of beer?

[fosterlynn]

in a exchange office, the resources we manage are currencies, even physical fiat and digital fiat, social currencies, etc... how can we track them in ocp? as resource types, or?

Currencies seems to be confusing, but if you are tracking quantities of currencies, like how many FairCoins do you have, they are resources.

Agree. The resource type of that resource is not FairCoin, it is something like FairCoin Address. And the resource itself is some address in a wallet. Or for physical fiat, the resource can be a bank account, or a virtual account that is part of a bank account, or a specific box of cash under the counter. The resource type then would be Bank Account or Virtual Account or Cashbox. Does that make sense?

[bum2]

Liters of milk? Liters of beer?

Doesn't matter the type of 'liquid' in this case, if it is useful for a project to see totals in that unit (because they used it), they should be able to choose it and see totals using any unit type (not only value)... or see their faircoins balance in other units, etc. It is just about having enough future flexibility, and use the ratio converter tables between units, etc. Maybe a transport project want to track kilometers or whatever unit desired...

[bhaugen]

Sure. But now you are talking about conversions between units of measure of the same resource.

I'm trying to establish the difference between units of measure, and resources which might be measured in units of measure. Like liters of beer, vs ounces of beer. Or liters of milk, vs quarts of milk. But there is no mathematical formula that will convert liters of milk to ounces of beer.

[bum2]

Agree. The resource type of that resource is not FairCoin, it is something like FairCoin Address. And the resource itself is some address in a wallet. Or for physical fiat, the resource can be a bank account, or a virtual account that is part of a bank account, or a specific box of cash under the counter. The resource type then would be Bank Account or Virtual Account or Cashbox. Does that make sense?

The recipients of the currencies are resources of course (accounts, boxes), but the coins inside are also resources, even physical or virtual, so a balance is an amount of resources of some type, stored in another resource which can be a private account or fairaddress... if we move coins between agents we are moving resources of certain type, with a related unit to define value...

But there is no mathematical formula that will convert liters of milk to ounces of beer.

of course you can't convert between some different branches of units, mainly from same branch only or what makes sense... (hectares-to-squaremeters, tons-to-kilos, etc)

[fosterlynn]

... but we digress... :smile:

[bhaugen]

the coins inside are also resources

Yes, and that seems to be a cause of confusion. Faircoin is (in OCP) both the name of a unit and the name of a resource type. But those are different concepts, even if they have the same name. It's like sex and sex: one is a gender, the other is an enjoyable activity that might result in babies.

of course you can't convert between some different branches of units, mainly from same branch only or what makes sense...

Those are not different branches, beer and milk are different resource types.

[bum2]

... but we digress... :smile:

Totally true... let's focus the splitting issue... for me the point was to think about common data in order to see the needed splittings in the code, so common apps can share also common data on them for all 'forks'...

[bum2]

But anyway, the work required to split the code is not for this next release, so first i want to focus on the subject explained in the docs... if we add or not the General app and the types trees, and to which extend we want to use or not other structures offered by that app, like the relations or the unit-ratios, the regions, etc. If we achieve a working release that covers some of this and we are all happy with that, then we can think again about how it makes sense to split the code...

[bhaugen]

Don't get me wrong, I like the General app. But I also have worked with REA for 20 years, and it has been beaten up around the world for 30+ years, and is an international standard, and it has proven to work. So if the General ontology and the REA ontology want to co-exist in the same system, they need to figure out how they work together, and what is what in terms of concepts and relationships. And what is a unit and what is a resource type.

[fosterlynn]

@bum2 do you have an estimate on when this release that includes the general app will be ready? It also needs to be tested by FC admin people, who show no sign of testing it. Even enough to discuss if they agree it is a requirement.