Create a data model for the info required to collect "metadata" from FOIA offices

[pkarman]

The starting point is in the schema document.

Examples are available.

The assumption for the data model is that an agency may have multiple components (the decentralized FOIA office model, e.g. USDA) or a single component (the centralized model, e.g. GSA). Either way, it makes some sense for the agency-to-component relationship to be a one-to-many.

• [x] Data model includes 90% of the universal FOIA request form fields (not unique to individual office)
• [x] All fields from agency YAML are present in the data model
• [x] Data model includes submission methods and request form data

Note: Removed the 4th acceptance criteria (the build-out in Drupal), since there are now discrete issues for those (#40 and #41)

[ba66e77]

What we have so far is a FOIA Office content type (screenshot of configured fields is below) which corresponds to what was referred to as a component above. The FOIA Office is associated to a Department/Agency in many-to-one manner. The FOIA Office also has an entity reference field (called the Request Submission Form) to a webform entity. This is a one-to-one relationship.

The Department/Agency is a just a taxonomy term with a name, description, and an image upload for the agency seal. Additional fields can be added if needed. An example of a Department/Agency as rendered without any styling work is shown below.

[ba66e77]

@pkarman @bsweger, could one of your crew compile the field list needed from what's in the schema so we have all the details needed in the ticket?

[pkarman]

Here's an example of the USDA OIG component. This is pulled from https://raw.githubusercontent.com/18F/foia-recommendations/master/prototypes/form-wizard/assets/js/agencies.js

{
"Office of the Inspector General, Department of Agriculture ( USDA )" : {
      "name" : "Office of the Inspector General",
      "fax" : "202-690-6305",
      "emails" : [
         "alison.decker@oig.usda.gov"
      ],
      "summary" : {
         "abbreviation" : "USDA",
         "description" : "The Department of Agriculture provides leadership on food, agriculture, natural resources, and related issues.",
         "name" : "Department of Agriculture",
         "usa_id" : "49015"
      },
      "address" : {
         "street" : "1400 Independence Avenue, SW",
         "city" : "Washington",
         "address_lines" : [
            "Alison Decker",
            "FOIA Officer",
            "Room 441-E Whitten Bldg."
         ],
         "state" : "DC",
         "zip" : "20250"
      },
      "phone" : "202-720-5677",
      "request_time_stats" : {
         "2011" : {
            "complex_highest_days" : "489",
            "complex_average_days" : "47",
            "simple_highest_days" : "305",
            "complex_median_days" : "20",
            "expedited_processing_median_days" : "0",
            "expedited_processing_average_days" : "0",
            "simple_average_days" : "22",
            "expedited_processing_lowest_days" : "0",
            "simple_lowest_days" : "2",
            "complex_lowest_days" : "1",
            "expedited_processing_highest_days" : "0",
            "simple_median_days" : "16"
         },
         "2010" : {
            "complex_median_days" : "100",
            "expedited_processing_median_days" : "0",
            "expedited_processing_average_days" : "0",
            "simple_average_days" : "14",
            "expedited_processing_lowest_days" : "0",
            "expedited_processing_highest_days" : "0",
            "simple_lowest_days" : "2",
            "complex_lowest_days" : "41",
            "simple_median_days" : "8",
            "complex_highest_days" : "778",
            "complex_average_days" : "156",
            "simple_highest_days" : "34"
         },
         "2014" : {
            "simple_median_days" : "61",
            "complex_lowest_days" : "42",
            "simple_lowest_days" : "2",
            "simple_average_days" : "85",
            "complex_median_days" : "141",
            "simple_highest_days" : "413",
            "complex_average_days" : "214",
            "complex_highest_days" : "682"
         },
         "2013" : {
            "simple_median_days" : "22",
            "simple_average_days" : "25",
            "simple_lowest_days" : "6",
            "complex_lowest_days" : "33",
            "complex_median_days" : "42",
            "complex_average_days" : "46",
            "simple_highest_days" : "79",
            "complex_highest_days" : "283"
         },
         "2012" : {
            "complex_average_days" : "68",
            "simple_highest_days" : "30",
            "complex_highest_days" : "133",
            "simple_median_days" : "13",
            "simple_average_days" : "13",
            "complex_lowest_days" : "31",
            "simple_lowest_days" : "1",
            "complex_median_days" : "64"
         }
      },
      "top_level" : false
   }
}

The request_time_stats are tricky. Right now they are culled from the annual report data in order to provide some meaningful contextual information to requesters for setting expectations. We don't want to introduce more data-entry burden on agencies when they are already submitting annual report info to OIP. We should talk with OIP about process and which annual report data to require here and what we might be able to automate from their existing process.

There are new fields for submissions methods and the request form that @bsweger and I will need to articulate here. Look for a few comments on this ticket as we aggregate.

[pkarman]

A note on the screenshot you've posted @ba66e77.

The fields for expedited processing and fee waiver instructions feel like they belong on the request form model, not at this office level. I don't have strong feelings about the model per se, but the presentation context of those fields to the agency officers feels like it belongs to the form, not the agency itself.

[pkarman]

Here's another example. Note that keys like reading_rooms, request_form, service_center, foia_officer and public_liason are present here and not on the previous example.

{
"National Finance Center, Department of Agriculture ( USDA )" : {
      "name" : "National Finance Center",
      "reading_rooms" : [
         [
            "Click here to access the NFC Electronic Reading Room",
            "https://www.nfc.usda.gov/FOIA/foiaereading.html"
         ]
      ],
      "summary" : {
         "usa_id" : "49015",
         "name" : "Department of Agriculture",
         "description" : "The Department of Agriculture provides leadership on food, agriculture, natural resources, and related issues.",
         "abbreviation" : "USDA"
      },
      "emails" : [
         "cheri.alsobrook@nfc.usda.gov"
      ],
      "fax" : "504-426-9706",
      "service_center" : {
         "phone" : [
            "504-426-0300"
         ],
         "name" : "Sandy Francois"
      },
      "website" : "https://www.nfc.usda.gov/FOIA/FOIA_home.html",
      "address" : {
         "city" : "New Orleans",
         "street" : "P.O. Box 60000",
         "zip" : "70160",
         "state" : "LA",
         "address_lines" : [
            "Cheri Alsobrook",
            "FOIA Officer"
         ]
      },
      "public_liaison" : {
         "name" : "Shri Alsobrook, Sandy Francois",
         "phone" : [
            "504-426-0327"
         ]
      },
      "request_form" : "https://efoia-pal.usda.gov/palMain.aspx",
      "top_level" : false,
      "foia_officer" : {
         "name" : "John Hemstreet",
         "phone" : [
            "504-426-0168"
         ]
      },
      "phone" : "504-426-0374"
   }
}

[pkarman]

The things we don't want from https://github.com/18F/2015-foia/blob/master/contacts/data/USDA.yaml are the keywords and (maybe) request_time_stats.

[ba66e77]

@pkarman we debated the placement of the fee waver and processing fields. In the end, we put them on the office because we didn't see any instances where the form was presented without the office context and putting them on the office made the content editing UI a little nicer. If that first observation is wrong or there are other factors we're not aware of then we can change the placement.

[bsweger]

@pkarman and @ba66e77 ... am compiling a single field list from the various links here and have a few questions to make sure I'm capturing things correctly.

1. For decentralized agencies should I assume that we want to capture the hierarchical relationship between a department/sub-agency and it's main agency (e.g., animal and plant inspection --> USDA)?
2. If yes to above, do we know how many levels deep the hierarchies go?
3. For decentralized agencies, are FOIA requests sent to the sub-agencies only, or can they be sent to both a top-level agency and a sub-agency depending on the nature of the request?
4. The 2015 USDA YAML has a misc field that contains the name of the FOIA director, for example. Are we looking for our schema to have that kind of flexibility, or do we want to explicitly model all needed fields? Another example is the notes field.

[bsweger]

@ba66e77 Following up on the fee waiver and processing fields, am I thinking correctly that these will vary based on the specific FOIA requests? If that's true (and please let me know if it's not!), these do seem like attributes that should be defined as part of the request schema.

I don't have strong UI feelings, since I haven't participating in any testing to date. But we're just talking logical model/schema here?

[pkarman]

For decentralized agencies should I assume that we want to capture the hierarchical relationship between a department/sub-agency and it's main agency (e.g., animal and plant inspection --> USDA)?

yes. IIUC, that's represented in the screenshot by the "entity relationship" to the "Agency/Department". I.e. an office can have a parent office.

If yes to above, do we know how many levels deep the hierarchies go?

It goes one level deep.

For decentralized agencies, are FOIA requests sent to the sub-agencies only, or can they be sent to both a top-level agency and a sub-agency depending on the nature of the request?

A single request goes to a single office (component). If a requester needed to get records from multiple components within a single agency, they would need to file one request for each component.

The 2015 USDA YAML has a misc field that contains the name of the FOIA director, for example. Are we looking for our schema to have that kind of flexibility, or do we want to explicitly model all needed fields? Another example is the notes field.

I like flexibility, but I like to use it to discover patterns that the model doesn't yet have explicit slots to represent. So if folks have been using misc to capture FOIA director, I would prefer to keep the misc field and add a foia_director field.

[pkarman]

@bsweger The fee waiver and expedited processing instructions are static text that will be displayed on the request form, to help the requester understand how much a FOIA request might cost. It's help text. So it doesn't change from request to request, but does change from office to office (possibly) and from agency to agency (likely).

[bsweger]

@pkarman aaaah---thanks for the info re: fee waiver/processing....missed that they are static instructions.

[bsweger]

@pkarman Thanks for the info re: the hierarchies.

A single request goes to a single office (component). If a requester needed to get records from multiple components within a single agency, they would need to file one request for each component.

I phrased my original question poorly...let me try again. For an agency like USDA, would we expect that USDA itself (i.e., the top level of a decentralized agency) receive FOIA requests? Put another way, can a requester choose to route a request to the top level of a decentralized agency? Or is the choice limited only to the 2nd tier departments?

I'm asking because in the sample USDA metadata, there's a series of departments with all of the fields needed to created a request. Would we expect to have this information defined for USDA proper as well?

Does this question make sense?

[pkarman]

It does make sense @bsweger thanks for clarifying.

Each FOIA office can receive requests. So the main office in a decentralized agency can receive requests like any of their components.

[bsweger]

Ok, here's a first pass at a complete list of fields. The names aren't meant to represent what we should use; I added the prefixes to better show, in this flattened format, the 1:n relationships between a department and some of the descriptive names needed to render the form.

This exercise raised a few questions about data structures/models that we'll use to store the info, but perhaps I'm jumping ahead. Would love to use some of our sync time today to go through this and better understand how schemas are represented in the current Drupal implementation and how we can capture their change history, since we'll no doubt have to iterate as we get feedback.

Also @pkarman great point about the stats; agree that we need some thinking there too.

Field Name	1:n with department?
name
fax
emails	y
summary_abbreviation
summary_description
summary_name
summary_usa_id
address_street
address_city
address_lines
state
zip
phone
top_level_flag
reading_rooms	y
service_center_phone
service_center_name
website
request_form
foia_officer_name
foia_officer_phone
public_liaison_name
public_liaison_phone
submission_format	y
submission_name	y
submission_title	y
submission_address_lines	y
submission_email	y
submission_url	y
submission_fax	y
required_form_fields_name	y
required_form_fields_label	y
required_form_fields_regs_url	y
required_form_fields_help_text	y
required_form_fields_enum	y
additional_form_fields_name	y
additional_form_fields_label	y
additional_form_fields_help_text	y
request_stats_year	y
request_stats_complex_highest_days	y
request_stats_complex_avg_days	y
request_stats_complex_lowest_days	y
request_stats_complex_median_days	y
request_stats_simple_highest_days	y
request_stats_simple_avg_days	y
request_status_simple_lowest_days	y
request_stats_simple_median_days	y
request_stats_expedited_highest_days	y
request_stats_expedited_avg_days	y
request_stats_expedited_lowest_days	y
request_stats_expedited_median_days	y

[ba66e77]

@bsweger, it seems like there are two conceptual entities here that the list is trying to represent as a single entity. To my thinking, Agency is an entity and Office/component is an entity, so we should be developing two schemas here. The Agency would contain an array of Office entities to reflect the one-to-many, has-a relationship between the Agency and its Offices.

The yml files I've seen for agencies reflect the structure you've outlined here, but what do you think of something structured more like the below?

name: Department of Energy
ID: xxxxxxxx
description: Morbi pulvinar ante vitae porttitor.
seal: doe.jpg
offices:
    office
      name: Headquarters
      email: ....
      foia officer: ...
      website: .....
      ....other office properties...
    office
      name: Chicago office
      email: ....
      foia officer: ...
      website: .....
      ....other office properties...

[bsweger]

@ba66e77 Yes, for sure. The table above is a flat list of all the fields I think we'll need. Modeling the relationships seems like the next logical step. In addition to the one to many between department and offices, there are also one to many relationships between and office and submission format, etc.

Would be helpful to schedule some time with @pkarman or use a meeting already on the calendar to flush out some acceptance criteria for this story? When we say "create an agency data model" are we talking about the json/yaml representation? A logical relational database model?

[ba66e77]

I think what we're talking about is a pseudo-code representation. YAML, JSON, bullet-points, whatever. So long as we don't end up conflating defining the model with how the model will be communicated between the front end and the back end.

[pkarman]

@ba66e77 good point about not conflating how the data is stored vs how it is serialized.

Thanks @bsweger for putting together all the marbles into one list. I agree; some talking time to make sure we're all aligned about how to represent the relationships seems fine. I agree that my mental model so far has been Agency and Offices, where an Agency might have one Office (centralized) or multiple (decentralized).

[pkarman]

I also realize that we have used departments in some examples where in others we have used offices or components. Those all refer to the same thing: the place where a FOIA request is submitted. I think we should standardize our language. OIP refers to components and offices most often. My personal preference would be for offices.

[bsweger]

As I'm putting these fields into into a more hierarchical format, I still have questions about departments and offices (agreed that office is preferable to component).

I know I'm beating a dead horse here, but if an office is the entity that receives a FOIA request and a FOIA request can also be made at the department/agency level, even in a decentralized agency, would we expect to see main agencies listed as offices?

For example:

Centralized Agency

Department Name: GSA
Department attribute #1
Department attribute #2
Office:
  Name: GSA
  FOIA Officer attributes
  E-mails
  reading rooms
  submission methods
  etc.

Decentralized Agency

Department Name: USDA
Department attribute #1
Department attribute #2
Office:
  Name: USDA <-- note that main USDA is repeated here to be treated as an office that can receive FOIA requests
  FOIA Officer attributes
  E-mails
  reading rooms
  submission methods
  etc.
Office:
  Name: Agricultural Marketing Service
  FOIA Officer attributes
  E-mails
  reading rooms
  submission methods
  etc.

[pkarman]

You can't make a request at the agency level. I.e. in a decentralized model, you can't request records from across the entire agency (all offices) by submitting to a single main office. It's really up to the agency how they organize/name themselves, but each office must have a FOIA officer, by law, so it's the individual FOIA office to which you must send a request.

So in your example, yes, USDA might be listed twice, once as the name of the agency and once as the name of the main office. The naming of the office is up to them, and may or not be misleading. You can't submit a request to the agency because the agency itself doesn't have a FOIA office. [that's misleading. the agency's FOIA office is the main office] The main FOIA office is first amongst equals.

Note I have tried to use the word agency not department, since department seems to not be used by OIP at all.

[bsweger]

@pkarman you're right--there are way too many words for agencies, bureaus, etc. I'll use agency going forward to represent the top level.

@ba66e77 spoke with Peter b/c I needed a synchronous convo to clarify this point about the main FOIA office in a decentralized agency. It turns out that I was thinking of an agency and it's main FOIA office as one in the same. Once Peter set me straight on that, things became much clearer.

Next stab at the fields forthcoming...

[bsweger]

@pkarman @ba66e77 Ok, here's an "unflattened" list of the complete universe of attributes, as I understand them right now. I'm still unable to suss out some of the relationships from reading through the various YAML/JSON examples, so still have a few questions:

1. Can a FOIA office have more than one FOIA officer at a given time?
2. Can a FOIA office have more than one public liaison at a given time?
3. Can a FOIA office have more than one contact e-mail addy at a time?
4. Can a FOIA office have more than one reading room link?

Wanted to ask the above explicitly because some of the JSON/YAML artifacts indicate that the answer can be yes, but wouldn't that be pretty confusing from a requester perspective?

The format below assumes that required_form_fields and additional_form_fields are applicable across a FOIA office rather than descriptors of a particular submission format (e.g., online or e-mail).

Also pinging @adborden for his 👀 , since he was involved in this during discovery.

Look forward to hearing what I missed, etc.

agency: United States Department of Agriculture
abbreviation: USDA
usa_id: 12345
offices:
  office
    id: 99999
    name: United States Department of Agriculture
    description: USDA Main FOIA office
    email: usda_foia.usda.gov
    website: https://usda_main_foia.gov
    foia_officers:
        - name: Alice Jones
        phone: 202-222-3333
        - name: Pete Metaphor
        phone: 202-888-9999
    service_centers:
        - name: Scott Kirk
        phone: 202-123-4567
        - name: Michele Janeway
        phone: 413-444-5555 
    public_liaisons:
        - name: Jim McCoy
        phone: 202-123-4444
        - name: Sue Simile
        phone: 202-666-7777
    reading_rooms: 
        - url: http://foia_reading.usda.gov
        - url: http://more_foia_reading.usda.gov
    required_form_fields:
        - name: request_origin
          label: Request Origin
          help_text: (for example, New England Region 1A)
          regs_url: null
          enum:
          - Company
          - Individual/Self
          - Organization
    additional_form_fields:
        - name: contract_number
          label: The Contract Number
          help_text: If your request relates to a GSA contract, please provide the contract number
    submission_methods:
        - submission_format: paper
          name: Bob Smith
          title: FOIA officer
          address_lines:
            - Bob Smith
            - Room 123-B
            - A Building Name
          street: 333 Massachusetts Ave
          city: Washington
          state: DC
          zip: 02025
        - submission_format: email
          email: foia@usda.gov
        - submission_format: web
          url: https://submit_foia_form/form.aspx
        - submission_format: fax
          fax: 202-123-9876
    request_stats:
        - year: 2016
          complex_highest_days: 222
          complex_avg_days: 222
          complex_lowest_days: 222
          complex_median_days: 222
          simple_highest_days: 222
          simple_avg_days: 222
          simple_lowest_days: 222
          simple_median_days: 222
          expedited_highest_days: 222
          expedited_avg_days: 222
          expedited_lowest_days: 222
          expedited_median_days: 222

[pkarman]

This is great @bsweger thanks.

I think the answer to all your questions is "yes" but this feels like a great topic to talk with OIP about to clarify.

[bsweger]

Ok, will ask OIP about those questions.

Had a very high-level convo with Matt to your earlier point, @pkarman:

The request_time_stats are tricky. Right now they are culled from the annual report data in order to provide some meaningful contextual information to requesters for setting expectations. We don't want to introduce more data-entry burden on agencies when they are already submitting annual report info to OIP. We should talk with OIP about process and which annual report data to require here and what we might be able to automate from their existing process.

If I understood OIP correctly:

1. agencies are responsible for aggregating the annual statistics (via a spreadsheet w/ macros) from all of their individual FOIA offices
2. those aggregated numbers are sent to OIP for review by compliance folks
3. once OIP has reviewed the agency data and signed off, they provide a copy of the final, human-readable numbers back to the agencies, who use them to create the annual reports they're required to post online
4. OIP also generates an XML version of those finalized, agency-level, human-readable numbers. This XML goes to the person who maintains the Tomcat system.

It sounds like previously there wasn't necessarily a use for the more granular office-level stats once they'd been used to derive the agency-level info. But now there are two additional things to consider:

1. the new statutory requirement that FOIA offices post a machine-readable version of their raw stats data
2. the discovery story that needs to surface the office-level numbers to requesters

Which is just to say that you're right--we need more thinking here. I'll leave the stats fields at the office level in the list above, but we may also need to capture them at the agency level.

[bsweger]

Updated the list above to reflect the possibility of multiple foia officers and public liaisons. Remaining questions:

1. @ba66e77 and @pkarman were you all hearing that there could also be multiple service centers for a FOIA office?
2. Re: the reading room vs other places that agencies post records...those feel like different things to me, but that's possibly (likely) me splitting a data modeling hair. Does it make sense to make reading_room a list of URLs that can accommodate all link to FOIA records or info? Or do you think we should be explicit about naming these things as different attributes (e.g., reading room vs online records)?

[pkarman]

there could also be multiple service centers for a FOIA office?

I didn't hear that explicitly, but in general, for things like this where we know edge cases will likely exist, I usually prefer to assume multiples of everything.

Does it make sense to make reading_room a list of URLs that can accommodate all link to FOIA records or info?

Yes to that. Don't think we need more than URLs. It basically serves as a "see also" link for requesters.

[ba66e77]

I'm in agreement with @pkarman on both points above

[bsweger]

@ba66e77 @pkarman sounds good--updated that latest field list to make the possibility of many "reading room" urls and service centers.

Looks like the next thing to do is tackle that 90% item in the AC--will check in about the Monday AM.

[bsweger]

@pkarman Do you have any thoughts on how to start tracking down these "90% of fields from universal FOIA requests"?

[pkarman]

@bsweger I think OIP indicated they could help provide. I would start by looking at: https://foiaonline.regulations.gov/foia/action/public/request/createRequest https://foia.state.gov/Request/Submit.aspx https://www.dhs.gov/dhs-foia-privacy-act-request-submission-form

[bsweger]

Ah thanks, @pkarman. I remember these links from Friday's convo but wasn't connecting that directly to the 90% part of this story. @ba66e77 How would you like to handle compiling this additional list of fields? Does it make sense for one person to get OIP input and the other to start culling fields from the links above? Or will that be duplicative?

[ba66e77]

My understanding was that Matt was going to compile the 90% of fields, so I'd wait for him to provide input.

But that 90% is the fields agencies collect on their Request Submission Form rather than the metadata about the agency itself. I think we should proceed with building out the Office and Agency content types with the fields you list above and treat the inclusion of the 90% into the template for Request Submission forms as a separate piece of work.

[ba66e77]

A couple questions about the field list above.

1. For submission format web, do we need to capture this? That's the url of where their existing form is, but once we implement this new system would we still display the link for their old form?
2. For "required form fields" and "additional form fields", those relate to the office's request submission form, correct? Assuming so, they don't seem like they should be reflected directly within the office structure to me.

[pkarman]

For submission format web, do we need to capture this? That's the url of where their existing form is, but once we implement this new system would we still display the link for their old form?

Yes, we should continue to store the link for their "old" form. I can think of examples where agencies would prefer to use their existing form until such time as their case management system has direct API integration, since using their form saves them data entry work.

For "required form fields" and "additional form fields", those relate to the office's request submission form, correct? Assuming so, they don't seem like they should be reflected directly within the office structure to me.

Agreed. I think those refer to the Request Submission Form data, which we'll capture under a related entity separate from the Office.

[bsweger]

Update: OIP confirmed at today's sprint planning that the 90% list is in progress

[bsweger]

@ba66e77 Yes, good point re: keeping the request submission form data separate. Are you suggesting, then, that we create a separate issue for that and move the acceptance criteria? Or should we continue to work this is a single ticket and just internally differentiate agency/office and request submission form?

[adborden]

~Minor point: I think we should leave out "foia officer". These are essentially staff of the FOIA office. OIP has already said that agencies would be reluctant to provide that level of contact info. FOIA officers would not be able to answer questions unless they were directly related to a case they were working on. All contact should be done through the FOIA Service Center. The FOIA Liaison supervises the Service Center and helps to resolve disputes.~

Edit: got clarification today from OIP, FOIA Officer does exist in contact information for some agencies and is OK to include here.

[ba66e77]

@bsweger, to me, the Office and the Request Form are two separate entities and should be handled as separate pieces of work.

Actually, the same is true of the Agency. An Agency has an Office and and Office has a Request Form, but the Agency can live without the Office and the Office can live without the Request Form.

I would then have three tickets to build out the three objects and a fourth to represent them all as one unified JSON file.

[bsweger]

@adborden oh, interesting point about foia officer. To make sure I understand: are you suggesting that we remove FOIA Officer as an attribute of FOIA office but still continue to capture the name/title of a FOIA officer as part of individual submission_methods as needed?

In other words, for the above example, foia_officers Alice Jones and Pete Metaphor would not show up, but FOIA officer Bob Smith would be present as an attribute of the paper submission method?

[bsweger]

the Office and the Request Form are two separate entities and should be handled as separate pieces of work.

Actually, the same is true of the Agency. An Agency has an Office and and Office has a Request Form, but the Agency can live without the Office and the Office can live without the Request Form.

@ba66e77 Thanks for weighing in here. Your point gets at something that I've been struggling with myself. The data modeling that we're doing in this issue is ultimately what's needed to support the self-service metadata collection for a FOIA office (the resulting metadata in turn supports being able to process an individual request, which, as you note, isn't part of this ticket).

If I'm understanding correctly, 3 categories of info are necessary for the metadata collection:

1. agency
2. office
3. a potential universe of questions that an office might need to ask a FOIA requester (_i.e., the "90%" list from OIP)

Agreed that from a modeling perspective, the three things above should be treated separately, and we can split up that work however we see fit. But I want to check my assumption that no. 3 is a required part of the metadata effort. Is that right?

[ba66e77]

As implemented at present, the JSONAPI endpoint for an Office allows you to also retrieve the details of that Office's Request Submission Form. But from a back in perspective we are not opinionated about whether it should be included in the output or not. Tell us what you want in the JSON file and we'll find a way to get it there. By structuring the Agency, Office, and Request form as separate entities we can then compose them as desired into the combined metadata form.

[adborden]

@bsweger conversation from today re: foia officer sounds like we should keep it in as is for now. I don't think their role is restricted to paper submissions.

[bsweger]

@adborden Since you were involved in creating the sample .yaml files in discovery, was wondering if you have any insight into these fields from the draft above:

required_form_fields:
        - name: request_origin
          label: Request Origin
          help_text: (for example, New England Region 1A)
          regs_url: null
          enum:
          - Company
          - Individual/Self
          - Organization
    additional_form_fields:
        - name: contract_number
          label: The Contract Number
          help_text: If your request relates to a GSA contract, please provide the contract number

Was the intent that required_form_fields and additional_form_fields represent information that an office needs to collect from a FOIA requester? In other words, do we expect that many of these items would be pulled from the "90%" list provided by OIP?

[bsweger]

@malikkotob Thanks for the pings that remind me to update this issue.

To be more explicit about where things stand:

• The structure in this comment represents our current understand of the agency-->office objects, including office-specific submission methods and office-specific FOIA questions (i.e., those that aren't represented on the "90%" list provided by DOJ. That should satisfy the 2nd and 3rd acceptance criteria.
• The "90%" field list is here: https://drive.google.com/open?id=0B4JtVmWTTdQEU0lraVdZNVB2d0k. That should satisfy the first acceptance criteria

In other words, it sounds like there might be 3 over-arching entities: agencies, offices, and the pool of common FOIA questions. Does that sound right?

Looks like you've opened new stories for getting this info into Drupal. Does that mean the fourth acceptance criteria here is no longer applicable?

[bsweger]

Oh, I forgot to add the caveat that I'd expect schema changes and updates to be part of the process. Particularly around the 90% of questions, since the research/design folks are actively testing those. If they learn that the current wording needs updated or that the office metadata schema needs to be expanded to cover things like adding an ability to customize the ordering of the common questions, we'll revisit.

But I think we're arrived at a good starting point.

[malikkotob]

Hi @bsweger! I was about to comment to say just that. I went ahead and split the implementation of those over-arching entities into their own actionable issues, so yes - we can go ahead and remove that fourth criteria from this ticket.

To split hairs with regards to the three entities, the entirety of the request form that an office uses to accept FOIA requests (the 90% that is common, plus the 10% that is unique) is the third entity we will capture in Drupal, as opposed to only the common 90%.

Also, no issue with with changes being part of the process. I think as those come up we can adjust in Drupal as needed, using issues in Github to track those changes.

I'll look over the 90% and the structure from your comment and let you know if I have any questions/comments. Nice work getting this along!

[bsweger]

@malikkotob Great, thanks. I'll remove the fourth acceptance criteria. Also, I moved this story into Review/QA on the board, since you'll be reviewing when you get back to the office.

[adborden]

I spoke to @bsweger offline about https://github.com/18F/foia/issues/16#issuecomment-319697728 but want to make sure to document the answer for everyone else:

Was the intent that required_form_fields and additional_form_fields represent information that an office needs to collect from a FOIA requester?

Yes. required_form_fields being what is required by regulation, addtional_form_fields being any other fields that an agency component would like to capture from the requester.

In other words, do we expect that many of these items would be pulled from the "90%" list provided by OIP?

Yes, but we can negotiate how this is implemented. The intent of the metadata file from the recommendations point of view is that the metadata file contains everything you would need to generate a request form for an agency and component. That implies that the 90% fields would be in the metadata file. It doesn't say anything about whether the 90% fields should be exposed in the backstage to FOIA officers.

From a data modeling perspective, we might want to capture the 90% fields as a separate entity from the request form entity. As mentioned in https://github.com/18F/foia/issues/16#issuecomment-320029595, there might be some research that would require us to have some customization of the 90% fields and that data would land in the request form entity. As an example, in our experience map/backstage demo, there was a possibility that a FOIA officer could choose one of three options to render the name fields (single field, first/last, or prefix/first/last/suffix). That kind of thing would be captured in the request form entity.