Change delimiter in OwnerOperator

[tlageman-epa]

Definition of Ready

• [ ] Does this meet the INVEST criteria?
• [ ] Does this have a clearly defined user?
• [ ] Does this have an estimate?
• [ ] Does this have acceptance criteria?
• [ ] Does this have sufficient context for the specific functionality being developed (design, business rules, etc.)
• [ ] Does this have a dev link been provided in the design?
• [ ] Is this traceable to a feature, epic, or PI?

User Story

As a I want So that

Acceptance Criteria

Given When Then

Development Notes

Does this Story require 508 Testing?

• [ ] Implementing a new authoring tool that will produce code / content for the application
• [ ] Implementing a 3rd party widget that will be part of the application
• [ ] Adding new / updating custom functionality
• [ ] Adding new / updating a page template
• [ ] Adding new / replacing existing functionality (unless the replacement has already passed 508 testing)
• [ ] Full Site Review prior to a scheduled EPA 508 review
• [ ] Planning a video that will appear on the application
• [ ] Producing a video that will appear on the application
• [ ] Producing Training materials
• [ ] Producing documents

Context

3/7/2023 (tlageman) - Current delimiter is a comma (,). This makes it impossible to parse out different Owner Operators from the outputted data. Previously, the old system had a "< br >" as the delimiter. Recommend using a "|" as we have for other parts of the new EASEY system.

This affects Facility/Unit Attributes, and all allowance and compliance data sub types and their streaming services counterparts.

Testing

Test Case: Tester:

Definition of Done

• [ ] Have all Acceptance Criteria been agreed to and validated by the scrum team?
• [ ] Has the code been unit tested and peer reviewed?
• [ ] Have the functional tests been executed?
• [ ] Have all defects been reviewed and dispositioned, resolved, or deferred?
• [ ] Is the user story ready to be deployed to the test/staging environments?

[shahbazkhan1999]

fixed the test cases and also update the node version along with delimiter changes https://github.com/US-EPA-CAMD/easey-streaming-services/pull/94

[ergjustin]

@shahbazkhan1999 Build is failed. Need to fix build error (update node version in github->workflow-> .yml files)

[ergjustin]

@shahbazkhan1999 Can you please add some local env testing screenshot.

[shahbazkhan1999]

1. Bottom of our app's main page first click on the button "Start your data query" (https://campd-stg.app.cloud.gov/)
2. Then move to the next page where we have data type select the data type "compliance" and also select the sub-data type "Allowance based"
3. After the selection of both click on the apply button and next apply each filter one by one in all filter '|' delimiter is applied check-in data table owner/operator column this delimiter is applied
4. Also download the CSV and check the owner/operator column.
5. I added some attachments for both CSV and Data Table views. Check these Attachments links below.

Private Zenhub Image

Private Zenhub Image Private Zenhub Image Private Zenhub Image

[tlageman-epa]

A few corrections @shahbazkhan1999 :

• account info - data preview is missing | and shows , still. The download looks good though.
• allowance holdings - data preview has the | but the download is still using the ,
• allowance transactions - data preview and download are still using the , Keep in mind that transactions has Owner (Transferor) and Owner (Transferee) fields that both need the | instead of the ,
• compliance emissions - data preview and download are still using the ,

[ergjustin]

@shahbazkhan1999 Allowance-> Account Information Download Not same as display Account Information Download

[alangmaid]

@shahbazkhan1999 examples of improper "|" in the csv remain. Compliance - Allowance Based. When the "Owner/Operator" data contain a "," the CSV incorrectly display a "|". The HTML correctly displays "," Private Zenhub Image I believe this bug occurs for ALL CSV files that include the Owner/Operator field including Facility data type. I.e., when the "Owner/Operator" name includes a "," it is incorrectly displaying as a "|". E.g., "PowerSouth Energy Cooperative | Inc. (Operator)" should display as "PowerSouth Energy Cooperative, Inc. (Operator)"

[alangmaid]

1) Allowance - Transactions preview AND CSV display a "," between names in the "Owner (Transferee)" and "Owner (Transferor)" fields instead of "|" For example, AEP Generation Resources, Inc.,Buckeye Power, Inc. should display as AEP Generation Resources, Inc. | Buckeye Power, Inc.

2) Compliance - Allowance based preview display a "|" in "Owner/Operator" when it should display a ",". For example, search on program = CSSO2G1 and preview. Ctrl-F "Springfield" Preview screen

[tlageman-epa]

ownerOperator is found in the following APIs/endpoints and should have the | delimiter applied between each owner/operator value in a record e.g., "Georgia Power Company (Owner)|Southern Company (Owner)|Alabama Power Company (Operator)" which should be returned in the API call, CAMPD UI, and CAMPD file (json or csv) download.

1. account-mgmt
   • allowance-holdings
   • accounts/attributes
   • allowance-compliance
   • emissions-compliance
2. facilities-mgmt
   • facilities/attributes
3. streaming-services
   • facilities/attributes
   • accounts/attributes
   • allowance-holdings
   • allowance-compliance
   • emissions-compliance

buyOwner & sellOwner is found in the following APIs/endpoints and should have the | delimiter applied between each owner/operator value in a record e.g., "Georgia Power Company (Owner)|Southern Company (Owner)|Alabama Power Company (Operator)" which should be returned in the API call, CAMPD UI, and CAMPD file (json or csv) download.

1. account-mgmt
   • allowance-transactions
2. streaming-services
   • allowance-transactions

Note in the account-mgmt API there are endpoints for each data type that end in /applicable or /owner-operators but I am not sure if these are effected by this work. I think not, but wanted to make sure they were known.

The above requirements should not interfere with the current functionality of the CAMPD UI and APIs where a user can input one or more owner/operator(s) and filter for records matching the inputted owner/operator(s). The filters currently exist for the:

• account-mgmt endpoints listed above
• streaming services endpoints listed above (except facilities/attributes)

@alangmaid

[alangmaid]

On the UI, the filters that include "Owner", Operator", or "Owner/Operator" that need to be verified as having the correct "|" treatment are (current verification status on staging in UI and download):

• Allowance, Account Information (correct)
• Allowance, Holdings (correct)
• Allowance, Transactions (correct)
• Compliance, Alllowance Based (correct)
• Compliance, Emissions Based (incorrect - Owner/ Operator displays "," instead of "|" between owner and operator. "|" displays as expected between multiple owners and multiple operators.)
• Facility, Facility/Unit Attributes (incorrect - Owner/ Operator displays "," instead of "|" between owner and operator. "|" displays as expected between multiple owners and multiple operators.)

[alangmaid]

Retagging this ticket to v1.4 to address remaining items which are to address the concatenation that occurs on the "Owner/Operator" columns in: Compliance, Emissions Based (Owner/ Operator displays "," instead of "|" between owner and operator. "|" displays as expected between multiple owners and multiple operators.) Facility, Facility/Unit Attributes (Owner/ Operator displays "," instead of "|" between owner and operator. "|" displays as expected between multiple owners and multiple operators.)

[lgiannini1]

@shahbazkhan1999 On the Compliance, Emissions Based and Facility, Facility/Unit Attributes Custom Data Downloads, Owner' Operator still displays a "," instead of "|" between owner and operator in the data preview and CSV files.

Compliance, Emissions Based:

Compliance, Emissions Based CSV Download:

Facility, Facility/ Unit Attributes:

Facility CSV Download:

[shahbazkhan1999]

Changes are already been made; I am just waiting for the PR to approve so create a new release and added @djw4erg as a reviewer

[lgiannini1]

Blocked by #6101

[ergjustin]

@lgiannini1

6101 I revert to previous (disable CSP). I think you can test from now onwards.

[lgiannini1]

@shahbazkhan1999 I am still seeing the issues noted above where a "," is displayed instead of "|" between owners and operators in the Compliance, Emissions Based and Facility, Facility/Unit Attributes Data Previews. The delimiter is displayed correctly in the Compliance, Emissions Based CSV Download, but not in the Facility, Facility/ Unit Attributes download

Compliance, Emissions Based Data Preview:

Facility, Facility/Unit Attributes Data Preview:

Facility, Facility/ Unit Attributes CSV Download:

[shahbazkhan1999]

@lgiannini1 changes were merged but not deployed I do not have the access for that but now they are deployed you can test.

[lgiannini1]

Delimiters look correct in Facility/Unit Attributes, and all Allowance and Compliance data sub types

[szintgraff]

Checked Owner/Operator Column in Facility, Allowance (Account Info, Holdings, and Transactions), and Compliance (Emissions and Allowance based) on Data Queries in staging: https://campd-stg.app.cloud.gov/data/custom-data-download. No comments on the facility, allowance, or allowance based compliance data.

For compliance data, emissions based: The Owner/Operator Column doesn't have a delimiter and seems to only have operator data in that column. Is that how the data is suppose to display? It is also that way in the production environment.

@lgiannini1

[szintgraff]

Moving to ready for deployment- Checked with Travis and the owner/operator column looks correct in the emissions based compliance.

[ergjustin]

Issue: 1 Data Type: Compliance Data Subtype: Allowance Based programCode : ARP facilityId: 2828

Issue : 2 The ownerOperator examples on the Swagger page use a comma instead of a pipe

[ergjustin]

Made changes in

1. easey-account-api: Removed extra pipe in ownerOperator and The ownerOperator examples on the Swagger page
2. easey-facilities-api: The ownerOperator examples on the Swagger page
3. easey-streaming-services : The ownerOperator examples on the Swagger page

[ergjustin]

Developer test

1. easey-account-api

2. easey-facilities-api
3. easey-streaming-services

[lgiannini1]

In the Facility, Facility/Unit Attributes CSV download, a "," is displayed instead of "|" between owner and operator.

On the streaming services swagger page, a "," is sometimes displayed instead of "|" between owners and operators

Result when using year=2021 and stateCode=IA:

Results when using year=2021 and facilityId=2828:

[ergjustin]

Fixed the above issue Developer test.

[lgiannini1]

Delimiters look correct on Facility/Unit Attributes, and all Allowance and Compliance data sub types in UI, CSV Downloads, and Swagger pages.

[tlageman-epa]

Took a look at this and agree with @lgiannini1 on each part except swagger pages. The data return is fine, but the example schemas need some tweaking to accurately reflect the delimiter patter of "value|value". Right now, some of them have "value , value" and "value | value". Here the corrections needed.

Account Mgmt

• Holdings - swagger page example schema - extra space around |
• Transactions - swagger page example schema - has , instead of |
• Attributes - swagger page example schema - extra space around |
• Account-based compliance - swagger page example schema - extra space around |
• Emissions-based compliance - swagger page example schema - extra space around |

Facilities Mgmt

• Attributes - swagger page example schema - extra space around |

Streaming Services

• Facility attributes - swagger page example schema - extra space around |
• Account attributes - swagger page example schema - extra space around |
• Holdings - swagger page example schema - extra space around |
• Transactions - swagger page example schema - has , instead of |
• Account-based compliance - swagger page example schema - extra space around | Emissions-based compliance - swagger page example schema - extra space around |

FYI @szintgraff

[ergjustin]

@tlageman-epa For the clarification : Transactions - swagger page example schema - has , instead of | Is it in the sellOwner ?

[tlageman-epa]

@ergjustin yes, buyOwner is fine

[ergjustin]

Developer Test Fixed

1. swagger page example schema - extra space around |

2. swagger page example schema - has , instead of |

[lgiannini1]

Example schemas on account-mgmt, facilities-mgmt, and streaming-services swagger pages have the correct delimiters for ownerOperator with no additional spaces.

[szintgraff]

@spetros-do

I checked facility, allowance account (account info, transaction, holding) and compliance (allowance and emissions based). I found one issue outlined below.

There is a pipe where I think a coma should be for "Louisiana 1". To find the screen shot below in facility data preview, I set the year = 2019, and facility = Louisiana 1. There may be other instances for that facility for example, I see it in 2020 and 2021 too. There are at least 14 instances of this incorrect pipe.

[ergjustin]

Till now owner/operator was split with “),” then it joins with “)|”. When the above situation “),” splits an extra pipe joins. The reason for that is owner/operator has (West),LLC which joins the pipe.

Solution: 05/09

const splitOprWithPipe = entity.oprDisplay?.split('|');
const uniqueOwn = [...new Set(splitOwnWithPipe)].join('|');
const uniqueOpr = [...new Set(splitOprWithPipe)].join('|');
const uniqueOwnOprList = [uniqueOwn, uniqueOpr];
const ownerOperator = uniqueOwnOprList.filter(e => e).join('|');

[szintgraff]

@mark-hayward-erg @ergjustin @alangmaid

This explanation doesn't align with what I understand about the approach for this ticket. Let's talk about it during the CAMPD weekly meeting later today. My questions are:

• Which database is this data coming from?
• What does it look like in the original database?
• Are there multiple operators in one column?
• If not, why are we splitting by ")," and then rejoining?

[lgiannini1]

Delimiters look correct on Facility/Unit Attributes, and all Allowance and Compliance data sub types in UI, CSV Downloads, and Swagger pages for ORIS 1391.

[szintgraff]

I tested the locations below and am moving to ready for deployment.

• Downloaded facility data from 2015-2021
• Compliance data allowance based for 2021 and 2022
• Compliance emission based data from 2015-2021
• Allowance, Account Info for all states
• Allowance, holding from 2015-2021
• Allowance transaction data from 2020-2021

Testing description: I spot checked unique occurrences of owner/operator columns in the following data pulls and specifically looked for "space|", "|space", "| INC", "| LLC", "| LP", "| LTD".