CityScope Unification/Standardisation

agrignard commented 4 years ago

This is an attempt to have a common place to discuss on the unification task. Nothing is set in stone but at least let's try to converge on a common place to discuss.

What we have so far (not pretending to have an exhaustive view so feel free to complete/comment)

So here the current entry point that should be used to unify everything and that should be included in the wiki if everyone agree:

wiki https://github.com/CityScope/cityscope.github.io/wiki/CityScope-Data-Format-version-2.0 (it has been moved from the cityIO repo to here). It is pretty weak for now but this is what we have (this is more or less the RS, RM, RL OS,OM,OL, etc) and it will hopefully quickly evolve as a synthesise of this issue in the following days
Other non github reference (for non github user), is the paper an overleaf initiated by @LAAP that should be also used as a reference https://www.overleaf.com/project/5dc19bd7180565000164bd25
Concerning cityIO https://github.com/CityScope/CS_CityIO/wiki/cityIO-Data-Structure
Concerning ABM Standard: https://github.com/CityScope/CS_Simulation_GAMA/wiki/GIS-Structure here the documentation of the input that a volpe like model is requiring to run https://github.com/CityScope/CS_Simulation_GAMA/wiki/Game-IT for what we call the Lyon Model
Concerning Macro Mobility https://github.com/CityScope/CS_Mobility_Service there is some standard here used to drive the ABM micro mobility model (both macro and micro) we can think on how to migrate it to this repo or at least make a link
Current github issues related to this topic: https://github.com/CityScope/CS_Simulation_GAMA/issues/66 https://github.com/CityScope/CS_cityscopeJS/issues/5

I am sure there is other entry point so please feel free to complete it.

doorleyr commented 4 years ago

Thanks @LAAP. A few comments below:

If we are going to have a density attribute, the name should make it clear what the units are. I suggest capacity_sqm. I also think this should be a required attribute.
Whether an attribute is static or dynamic really just comes down to the front-end design (and there may be multiple FEs using the same standards). For example, the same types definitions could be used in 2 projects with different interfaces. In both cases the "density" value specified is the default density for that type but in one project, the interface may allow the user to adjust this value. So for the purpose of defining the types, we don't need to differentiate between static and dynamic.
I think any economic attribute should be optional because (i) It's unclear at this point what economic values we are talking about and the important variables will differ by context and (ii) as Ariel mentioned, some projects may prefer to represent estimate economic values with modules.
Height and capacity are determined by physical design in a more direct way so it makes sense for the user to specify them directly.
there are still some formatting issues. eg. in some cases, you have key-value pairs where the key is an object and the value is a proportion. eg. "NAICS": {{"440000":"0.50", "450000": "0.50"}:"1"} this is invalid json because the key of a key-value pair must be a string. You can fix the above example by changing it to: "NAICS": {"use": {"440000":0.5, "450000": 0.5}, "proportion" :1} I know this is less tidy but unfortunately its necessary. I'm happy to help with correct formatting once the content is nailed down.

LAAP commented 4 years ago

Thank you @doorleyr ,

1) I see. Ok. I think that the "classes" can have a predetermined "capacity_sqm". That it will be an "average factor" that a "future Land use module" will multiply or divide by the different factors to meet the needs of each project.

2) However, with that premise, we could have also the "monetary_$" static average value, that gives the "sense" of if this land use is High or Low income. Isn't? Then, the "Economic module" will do a similar job than the defined by @doorleyr . isn't? So what if I give a "fixed" value from 0 to 1, being "0" Low and "1" High?

3)_"Height and capacity are determined by physical design in a more direct way so it makes sense for the user to specify them directly"... So "Height" is also predetermined. "height_m" (3m is the normal height for indoor space. It can be the "fixed" number

4) I have put the formatting issues as an Issue

LAAP commented 4 years ago

Hi all, this website is important for m2 per person in amenities: https://www.engineeringtoolbox.com/number-persons-buildings-d_118.html

LAAP commented 4 years ago

| CS Classes

Dear @all, Here I share a new iteration with the latest changes, please, let me know your thoughts Codes:

_"areappsqm"= m2 per person. Source = https://www.engineeringtoolbox.com/number-persons-buildings-d_118.html

"monetary$"_ = ranking from “0” to “1”. “0” low value asset “1” high value asset. Source = Educated guest

"floor#"_ = Where the land use is likely to be placed. “1” most likely at ground level, “2” most likely above ground level, “3” indifferent, both options are open. Source = Educated guest

HOUSING

1 | Household activities (Residential activities)

{
    " CS_Household_activities ": {
        "LBCS": {"use":{"1100"},"proportion":1}
         },
        "color": {#ff0000}
         },
        "NAICS": {"use":{"000000"},"proportion":1}
         },
       "areapp_sqm": {89}
         },
       "monetary_$": {0.75}
         },
       "floor_#": {3},
       }
 }

2 | Transient living (Residential activities)

{
    " CS_Transient_living ": {
        "LBCS": {"use":{"1200"},"proportion":1}
         },
        "color": {#ff6d6d}
         },
        "NAICS": {"use":{"000000"},"proportion":1}
         },
       "areapp_sqm": {65}
         },
       "monetary_$": {0.5}
         },
       "floor_#": {3},
       }
 }

3 | Institutional living (Residential activities)

{
    " CS_Institutional_living ": {
        "LBCS": {"use":{"1300"},"proportion":1}
         },
        "color": {#a70000}
         },
        "NAICS": {"use":{"000000"},"proportion":1}
         },
       "areapp_sqm": {50}
         },
       "monetary_$": {0.25}
         },
       "floor_#": {3},
       }
 }

AMENITIES

4 | Hotels

{
    " CS_Hotel ": {
        "LBCS": {"use":{"1300"},"proportion":1}
         },
        "color": {#fff402}
         },
        "NAICS": {"use":{"721110"},"proportion":1}
         },
       "areapp_sqm": {31}
         },
       "monetary_$": {1}
         },
       "floor_#": {3},
       }
 }

5 | Restaurants

{
    " CS_Restaurant ": {
        "LBCS": {"use":{"2500"},"proportion":1}
         },
        "color": {#ff0268}
         },
        "NAICS": {"use":{"720000"},"proportion":1}
         },
        "areapp_sqm": {1.86}
         },
        "monetary_$": {0.90}
         },
       "floor_#": {1},
       }
 }

6 | Night live

{
    " CS_Night_live ": {
        "LBCS": {"use":{"2540"},"proportion":1}
         },
        "color": {#e14511}
         },
        "NAICS": {"use":{"620000":"1"},"proportion":1}
         },
       "capacity_sqm": {4.64}
         },
       "monetary_$": {0.85}
         },
       "floor_#": {1},
       }
 }

7 | Leisure and Wellness

{
    " CS_ Leisure_and_Wellness ": {
        "LBCS": {"use":{"6500"},"proportion":1}
         },
        "color": {#f03087}
         },
        "NAICS": {"use":{"722500":"1"},"proportion":1}
         },
       "capacity_sqm": {2}
         },
       "monetary_$": {1}
         },
       "floor_#": {1},
       }
 }

8 | Culture

{
    " CS_Culture": {
        "LBCS": {"use":{"5000"},"proportion":1}
         },
        "color": {#d0d0d0} 
         },
        "NAICS": {"use":{"710000":"1"},"proportion":1}
         },
       "capacity_sqm": {9.29}
         },
       "monetary_$": {0.70}
         },
       "floor_#": {3},
       }
 }

9 | Shopping Centers

{
    " CS_Shopping_Centers ": {
        "LBCS": {"use": {"2100":0.75, "2500": 0.25}, "proportion" :1}
         },
        "color": {#9a0202} 
         },
        "NAICS": {"use": {"440000":0.50, "450000": 0.50}, "proportion" :1}
         },
       "capacity_sqm": {9.29}
         },
       "monetary_$": {0.70}
         },
       "floor_#": {3},
       }
 }

10 | Banks

{
    " CS_Bank ": {
        "LBCS": {"use":{"2200"},"proportion":1}
         },
        "color": {#bf6262}  
         },
        "NAICS": {"use":{"450000"},"proportion":1}
         },
       "capacity_sqm": {13.93}
         },
       "monetary_$": {1}
         },
       "floor_#": {1},
       }
 }

11 | Educational

{
    " CS_Educational ": {
        "LBCS": {"use":{"6100"},"proportion":1}
         },
        "color": {#008eb8}  
         },
        "NAICS": {"use":{"610000"},"proportion":1}
         },
"capacity_sqm": {2}
  },
"monetary_$": {0.6}
  },
       "floor_#": {3},
       }
 }

PARK

12 | Parks

{
" CS_Park ": {
        "LBCS": {"use":{"0000"},"proportion":1}
         },
        "color": {#00b828}  
         },
        "NAICS": {"use":{"712190"},"proportion":1}
         },
       "capacity_sqm": {1.02}
         },
       "monetary_$": {0.01}
         },
       "floor_#": {1},
       }
 }

TRANSPORTATION

13 | Transportation

{
" CS_Transportation": {
        "LBCS": {"use":{"4000"},"proportion":1}
         },
        "color": {#0044b8}  
         },
        "NAICS": {"use":{"480000"},"proportion":1}
         },
       "capacity_sqm": {0.60}
         },
       "monetary_$": {0.55}
         },
       "floor_#": {3},
       }
 }

SAFETY AND SECURITY

14 | Public Safety

{
" CS_Public_Safety ": {
        "LBCS": {"use":{"6400"},"proportion":1}
         },
        "color": {#2f79f6}  
         },
        "NAICS": {"use":{"480000"},"proportion":1}
         },
       "capacity_sqm": {0.60}
         },
       "monetary_$": {0.55}
         },
       "floor_#": {3},
       }
 }

WELLBEING

15 | Health Care

{
    " CS_Health_Care ": {
        "LBCS": {"use":{"6500"},"proportion":1}
         },
        "color": {#2fe7f6}  
         },
        "NAICS": {"use":{"620000"},"proportion":1}
         },
       "capacity_sqm": {7.43}
         },
       "monetary_$": {0.65}
         },
       "floor_#": {3},
       }
 }

OFFICE

16 | Public Administration

{
    " CS_Public_Administration_office ": {
        "LBCS": {"use": {"6200":0.50, "6300": 0.50}, "proportion" :1}
         },
        "color": {#74b7bd}  
         },
        "NAICS": {"use":{"920000"},"proportion":1}
         },
       "capacity_sqm": {10}
         },
       "monetary_$": {0.70}
         },
       "floor_#": {2},
       }
 }

17 | Finance

{
    " CS_Finance ": {
        "LBCS": {"use":{"2200"},"proportion":1}
         },
        "color": {#67353f}  
         },
        "NAICS": {"use":{"520000"},"proportion":1}
         },
       "capacity_sqm": {14}
         },
       "monetary_$": {0.99}
         },
       "floor_#": {2},
       }
 }

18 | Scientific and Technical

{
    " CS_Scientific_and_Technical ": {
        "LBCS": {"use":{"2400"},"proportion":1}
         },
        "color": {#d253c6}  
         },
        "NAICS": {"use":{"540000"},"proportion":1}
         },
       "capacity_sqm": {6}
         },
       "monetary_$": {1}
         },
       "floor_#": {2},
       }
 }

INDUSTRY

19 | Manufacturing

{
    " CS_Manufacturing ": {
        "LBCS": {"use":{"3000"},"proportion":1}
         },
        "color": {#df18ec}  
         },
        "NAICS": {"use": {"320000":0.50, "330000": 0.50}, "proportion" :1}
         },
       "capacity_sqm": {20}
         },
       "monetary_$": {0.95}
         },
       "floor_#": {1},
       }
 }

NATURE

20 | Nature

{
" CS_ Nature ": {
        "LBCS": {"use":{"9000"},"proportion":1}
         },
        "color": {#ffffff}  
         },
        "NAICS": {"use":{"712190"},"proportion":1}
         },
       "capacity_sqm": {28}
         },
       "monetary_$": {0.01}
         },
       "floor_#": {1},
       }
 }

RELNO commented 4 years ago

@LAAP thanks -- I'm confused as to why we went back to have arbitrary fields like "capacitysqm", "monetary$", "floor_#"? Especially after the significant effort put in finding citable, widely used standards like LBCS.

(just as a side note, using ascii terms like "#" in a json is allowed, but has better chance to break when parsing given)

LAAP commented 4 years ago

@RELNO ,

This is justa a new iteration. I think that "capacitysqm", "monetary$", "floor_#" are fields that needs to be "Dynamic". However, If I am understanding correctly the needs of the team, I think that some people in the team, like @doorleyr et al. are asking about having "values" in this fields that provides information about where in the type this Class will be allocated (1st floor, 2nd floor...), How many m2 per person does the class has, and short of a "economic" element.

With all that elements, I have made an iteration with "fixed weights" that a "future type-module" can just interpret and translate into his own understandable dynamic variables, but also, a more basic toy-model-table, can just use as a "placeholder" data for running. This placeholder data have a bit of sense, so this numbers are not totally "arbitrary", they are base in a lot of simplifications, average values, and assumptions, but I think that are ok for a toy model.

I suggest a Zoom meeting between all of us

RELNO commented 4 years ago

yes, might worth having a call on this. There's a difference between dynamic props (that could be anything, and will be added to GEOGRIDDATA object via user input), and what the new cityscope schema minimally requires.

LAAP commented 4 years ago

I totally agree with you @RELNO , and that was my first approach, and that is why I leave the GEOGRIDDATA object input as "[?]", because.

However I understood that @doorleyr suggested, some how, to have a fixed numbers.

Maybe it is a combination of both: we can have "capacity_sqm", "monetary_S", "floor_n" as dynamic props, and in parallel, we can have a "static-default library" with this "educated guest values" that, if the "type-module" doesn't work, or if the team implementing the models doesn't want to define their own type, they can use it.

RELNO commented 4 years ago

LBCS (SLUCM) to NAICS conversion

@LAAP @MarkusElKatsha: It looks like LBCS <=> NAICS conversion is a common practice (see refs below). If so, can we avoid doing this conversion ourselves and use any other the given resources below?

See: https://planning-org-uploaded-media.s3.amazonaws.com/legacy_resources/lbcs/background/QLBCSConvFunction2NAICS.TXT,

https://planning-org-uploaded-media.s3.amazonaws.com/legacy_resources/lbcs/background/pdf/rslucm2sic2naics.pdf. ,

https://planning-org-uploaded-media.s3.amazonaws.com/legacy_resources/lbcs/background/pdf/rslucm2sic2naicsnotext.pdf

LAAP commented 4 years ago

Thank you @RELNO very much for sharing,

Yes, we were aware of this correlation, and it has been used for defining the "simplified 20 classes". The mamut task was to "consolidate all this hundreds of "land uses" into only "20 classes".

In any case, we will put this links in the documentation, so people doing a more "granular" correlation and research, can have all the correlations.

Saying so, @MarkusElKatsha and I have a new proposal for "simplifying" the classes.

I will explained above:

LAAP commented 4 years ago

After a eficient talk with @MarkusElKatsha , we have got in to the following conclusions:

1) NAICS are not equal to Infogroups data

2) Infogroups' database provides a very rich information related to commercial elements: From here we where expecting to obtain the number of people in the amenity, and some sort of "tax related" economic data.

3) Unfortunately, Infogroup is NOT a open source database: You have to pay, and it is very specific for USA, we will not find a substitute in Europe, Latin America, Asia...

4) In a standardized and open CityScope we SHOULD NOT totally rely into Infogroup data, that has a "commercial access"

5) NAICS, by itself, only provides a more granular understanding of the "Land use"

6) Since, in this iteration of the tool, LBCS is already being simplified in 20 classes, seems to us that there is not need for a finer grain information like the one provided in NAICS

7) And Since, for Researchers working in a finer grain level like @cristian, the crosswalk between NAICS and LBCS, already exists (previous links shared by @RELNO ), and the crosswalk with our 20 types also has being made (see above) and will be documented, so they can make a compatible research with CityScope standards.

8) Since at this level of "consolidation" (20 classes) NAICS are kind of a redundancy, @MarkusElKatsha and myself have consider that it is NOT NEEDED to keep the NAICS as part of the "classes". LBCS is going to be enough. So the NAICS will be removed from the classes

9) Unfortunately, that forces us to have the 3 "dynamic" variables previously commented, that maybe manipulated with an slider (if they are enbeded in the front end) or defined in workshops and harcoded in the back end (TBD)

10) The 3 dynamic variables are: (1) sqm per person, (2) price of the sqm, (3) Floor

So the class could look similar to:

{
" CS_ Nature ": {
        "LBCS": {"use":{"9000"},"proportion":1}
         },
        "color": {#ffffff}  
         },
       "sqm_pp": {28}
         },
       "price_sqm": {0.01}
         },
       "floor": {1},
       }
 }

Please, let us know your thoughts

RELNO commented 4 years ago

considering https://cityscope.media.mit.edu/docs/modules/CityScope%20Types%20System it seems this issue can be closed. Specific unification issues can be opened as needed. Thanks @MarkusElKatsha @LAAP for the effort!

agrignard commented 4 years ago

Great effort everyone, one of the biggest issue ever open (on Nov 2019 ;-)) and fixed

The doc looks really great, good job! Next step will be to migrate existing projet or starts new one with this standardisation.

Are grassbrook and corktown already using it?

Happy to help for any specific part (for instance having a toy ABM structure running with this specification)

LAAP commented 4 years ago

Thank you @agrignard and @RELNO ,

I think that @MarkusElKatsha and I are ready to keep documenting guys... and I see some conversations going on with @crisjf regarding wikies' work... So let us know if you need us somewhere else.

CityScope / cityscope.github.io