unece-stat commented 9 months ago

Feedback relevant to the Design phase

Relationship Management

In “2.3 Design Collection”, there is a need to recognise the need to better understand and engage with respondents, so to mitigate the ‘new normal’ of challenges relating to response rates, collection operations etc. (eg. undertaking research to understand if incentives work). (New Zealand)
Lastly, can we add about following a Respondent Centred Approach to the design work? This is something the UN is promoting at the moment and are keen on encouraging other to follow – I am leading a task force with them on the method. Also it is in the survey playbook and others internationally are beginning to move to follow this method (e.g. stats NZ recently launched an RCD programme, ONS doing it, ABS, Austria and wider). The aim of the method is to drive up quality, reduce operational costs and in general help us to achieve our survey goals. Austria has found there are environmental benefits also to following this method. (United Kingdom)

Design of Dissemination

Sub-process 2.1: While Phase 7 (Disseminate) has many activities about release and dissemination, nothing about this is explicitly designed anywhere in Phase 2. It should be included as a task in sub-process 2.1, also in GSBPM explicitly. (GSBPM “Task” task team, 2022)
This sub-process is focused on the detailed design of the statistical product during the dissemination phase. Specific metadata, national and international standards, and practices are the inputs of this sub-process. This process includes Identifying mechanisms for providing access to detailed data through dissemination channels, Identifying the timing of the release, identifying information security procedures, Identifying the quality metrics that are supposed to be disseminated, and Identifying the statistical product promotion plan. (Saudi Arabia)

Design of Coding

We have introduced a new sub-process in phase 2 that covers the design of coding and editing, that puts an emphasis on a holistic view on these activities and where and when in the production to perform them. (Sweden)

Design of Evaluation

This subprocess includes designing and documenting the project success evaluation plan, including achieved outputs and used resources (e.g. data sources, registries, etc.), as well as the capabilities used to provide the sources of production activities (e.g. methods, processing, systems, standards, skills, etc.). It also includes this sub-process identifying the entities interested in the product for consultation and determining the timing and methods for collecting inputs and criteria to be used for evaluation, as well as the effectiveness of evaluating quality portals. And also identify, test, and validate monitoring, performance, and output criteria in this procedure to be implemented in the Eighth Phase “Evaluate”. (Saudi Arabia)

Design of Security (considered also for Build phase)

We suggest adding considerations on compliance with information security requirements concerning the design of production systems and workflow (point 55, p. 15) as well as mentioning security information implications regarding the testing of production systems (point 63, p. 16). We consider information security to be a crucial aspect of statistical production and we look forward to seeing it included within GSBPM. (Portugal)

Where does cartography sit?

Furthermore, for processes that involve data collection through surveys and censuses, it is necessary to count on cartographic materials and instruments (maps, plans, sketches and other related information), that enables the interviewers to locate themselves geographically and locate easily observation units. Therefore, it is suggested to include a sub-process regarding the design of statistical cartography or analyze in which sub-process could be included its activities and products, because in the new version it is not mentioned in any of its sub-processes. (Ecuador)

Non-linearity

When describing a process, interactions or loops may exist between sub-processes. It is described in the GSBPM document for the like of 2.2 and 2.5. It is very usefull to have more of such explanation in the specification document (France)

Boundary between Design and Build phases (and Overarching processes) (previously discussed in Build issues)

We have changed the balance between phases 2 and 3. We have moved most of the “build” phase to phase 2, to reflect that design and configuration of common tools is done at the same time. We have removed the building of IT tools from the statistical production process because we have a specific process for IT development. This means that we will only have plan (resources) and test the full production flow remaining in phase 3. (Sweden)
We suggest Phase 3 should be revised entirely. If the NSOs are using standard methods and IT tools for the business process, then the main activity here is to implement the plans prepared in Phase 2. Design using these tools. If a new tool/functionality needs to be developed, it is a separate process, not part of the data production. (Hungary)

Design vs Overarching processes

We will have some further overarching processes compared to the GSBPM, but these are still being developed and will be decided later. One of the overarching processes that we will have but that is not in the GSBPM is the design of statistical areas rather than the individual statistical programs, the results of which will be input to the production process of each individual program, with a clear connection to phase 2.2 that designs the program and its contents. (Sweden)

Boundary between Specify Needs and Design and phases

2.2: Many users find it difficult to understand the difference of level of conceptualization between 1.4 and 2.2 (France)
2.3: Many users find it difficult to understand the difference on agreements with the data providers in 1.5 and what there is still to be done in 2.3 (France)

Boundary between the Design and Collect phases (previously discussed)

It was previously decided that:

Based on a comment from Hungary, it was decided to move the bullet on Preparing a collection strategy from 4.2 to 2.3. This should also help to make 2.3 less instrument-focused. _(Comment:Phase 4……._There are elements in the description of sub-processes that belong to the Design phase and not collection (example: preparation of a collection strategy mentioned in sub-process 4.2). We suggest these should be moved to the right Phase. (Hungary) )
It was agreed to add something to 2.3 to mention design of fieldwork or operational planning, etc for survey collections, to address a comment of Mexico.(Comment:Add content to address the planning of the field work which may be a substantive part of this process when talking about surveys and censuses. This work includes tasks like calculating workloads and quantities of people needed, collection paths, and defining required characteristics of temporal offices, among other topics that are substantial for the process. This work is related to sub-process 4.2 but must be done at a previous stage as the resources to set up the collection must be negotiated previously to set up the collection. (Mexico)

Where are units defined?

Sub-process 2.2: “Statistical unit” is not discussed anywhere, but this is very important when we design variables, this needs to be in GSBPM explicitly. Also, this task comprises design of delineation of "statistical units" (observation and analytical units) from so called "units of collection" (administrative, legal etc. units). (GSBPM “Task” task team, 2022)
2.4: Identification of target population is not specific to surveys. Also is it in this sub-process that the statistical unit are defined ? (France)

Geospatial

We suggest adding a brief consideration concerning geospatial standards to point 46 as it follows: “Organisations are encouraged to reuse or adapt design elements from existing processes, and to consider geospatial aspects of data in the design through geospatial standards (or standards in the geospatial community/domain) to enhance the usability and value of the statistical information as well as the interoperability between both fields”. _2_ (Portugal)
We suggest changing the following sentence in the point 53 on design frame and sample: “It may include geospatial data and spatial analysis techniques that consider the spatial distribution of data collected, ensure geographical representativeness of the (geocode) samples and frames, and increase the efficiency of estimates. (…)”. This suggestion strengthens the geospatial component in this sub-process without going into too much detail like in the GeoGSBPM. 2.4 (Portugal)

2.1

2.1 : Add that this detail can be reviewed once the sub-process "apply disclosure control" is completed in 6.4 (France)

2.2

Phase 2.2 now has an emphasis on not only content variables, but also the context metadata of the statistical program. (Sweden, after their realignment)
2.2 Design variable descriptions: it includes the design of the final definitions, nomenclatures/classifications, variable descriptions: maybe renaming to “Design core metadata objects for the statistical business process” (or something similar) should be more forward looking. Also, the text mentions that metadata related to the data collection instrument needs to be designed here: maybe a more process-oriented approach is needed e.g. metadata for the entire statistical business process needs to be designed here. (Hungary)
Sub-process “2.2 Design variable descriptions” says: “It is expected that existing national and international standards will be followed wherever possible.” We suggest a change to say: “...wherever possible, to facilitate data integration among different statistical business processes.” (Mexico)
(See also France comment about 2.2 vs 1.4)
(See also question about definition of Units)
May be add that the linkage between variable is defined in the sub-process. Should be added the strong link between 2.2 and 2.5 for the creation of variables from these links (France)

2.3

2.3 Design collection: a proposal is to focus more on the mode design of the collection and the collection instrument in the text. Things come with a multimode design (sequence of modes, different methodological instruments for the different modes, etc.), while the collection instrument is only a part of it. (Hungary)
(See also France comment about 2.3 vs 1.5)
Collection design: This sub-process points out the most appropriate methods and instruments to collect data, from the different kinds of units of measurement to the different kinds of sources available; nonetheless, it is fundamental to include instruments design to backup and protect data and design of the process of extraction, transformation and loading (ETL) for the system of database management. (Ecuador)

2.4

(See also question about definition of Units)
Sub-process 2.4: The wording "population frame" is used to also cover non-sampling scenarios. Although GSBPM sub-process 2.4 says “This sub-process only applies to processes which involve data collection based on sampling, such as through statistical surveys.”, even if we use administrative registers (non-direct collection), we think there could be some kind of “filtering” of the entire register depending on the target population, also, for census, we will need some frame for the enumeration. (GSBPM “Task” task team, 2022)
Frame and sample design: This sub-process focuses on sampling-based statistics; nonetheless, and according to its own title, it should include as well frames that are designed for other kinds of data sources like censuses, for which, prior to data collection, a list of units of observation is obtained from cartographies and cadasters. (Ecuador)

2.5

Processing and analysis design: This sub-process addresses data processing and analysis design, therefore it is suggested to split the sub-process, one for processing and another for analysis, in order to differentiate the tasks and activities that involves each of them. (Ecuador)
2.5: Add the design of matching strategy in case of multi-sources (France)
We suggest changing the following sentence in the point 54 to “… This sub-process also includes design of specifications for collection tools to be used during the Collect phase and for data integration from multiple data sources, validation of data and estimation. Statistical disclosure control methods are also designed here if they are specific to this business process.” (Portugal)
2.5 Design processing and analysis: collection has its own design sub-process (2.3 with 2.4 strongly connected to it). 2.5 now says “design everything else”: there is no balance here. It is suggested to define design for each phase as separate sub-process or redesign the current version in a way that the scope of the sub-processes are more-or-less in balance. (Hungary)

2.6

Production systems and workflow design: Here is a need to specify this sub-process since it generates questions regarding whether its application is aligned with the process statutes that each producing entity internally manages or if it is necessary to have separate flows for each statistical operation (data source). (Ecuador)

Other previously-discussed feedback

Training machine learning (See Juan’s proposal): GSBPM does not contain anything on training machine learning models. Should this, and similar, model design and build activities be explicitly included or described in any phases? (United Kingdom)
Agile tool development (already addressed): Can we add about designing in an iterative and agile way? This would be mainly in the design and build phases – to ensure what we build is fit for purpose and not following waterfall development which risks building the wrong thing. (United Kingdom)

Chris-ECE commented 5 months ago

Suggestion: Consider comment under 2.1 to already be addressed on the basis that non-linearity of GSBPM applies : "Add that this detail can be reviewed once the sub-process "apply disclosure control" is completed in 6.4".