Save mappings for specific input file data sources (ESPM, Tax Lot, Building List)

RDmitchell commented 5 years ago

Expected Behavior

Implement saved mappings for defined input file data sources (PM, Tax Lot, Building List, etc) that the user would have control over through the UI. In other words, the user could define a mapping, save it with a name, and associate it with a file type (such as ESPM), if it was a file type that SEED knew about (from the import screen).

Alternatively, the user could select from the list of saved mappings when importing a specific file. They could have a saved mapping for their Tax Lot files and their Building List files, and when importing a specific file, they could choose which mapping to use.

Actual Behavior

n/a

Steps to Reproduce

n/a

Instance Information

n/a

nllong commented 5 years ago

Make sure to address the PM changing field names. That is, we need to version the PM field names as part of this work.

RDmitchell commented 5 years ago

look at #201 to make sure this addresses the concerns in that issue.

adrian-lara commented 4 years ago

@RDmitchell and @nllong

I'm hoping to identify an acceptable import file flow in terms of column mapping. Would the following work?

On the column mapping page of the import flow, there's a dropdown for the user to select a user-defined column mapping setting. Changing this option will require a confirmation click as current mapping "best guesses" by SEED or any of the user changes will be overwritten.

Once the column mapping setting is chosen, SEED populates the values according the mapping wherever there are exact matches between the setting and the import file column names. For anything that's not an exact match, SEED goes through it's "best guess" algorithm.

From here the process is similar to before where the user can make changes before continuing. They could also change the column mapping settings they chose to start.

adrian-lara commented 4 years ago

Per conversations, I will be coming up with more details and mocking out the above-mentioned flow. I'll provide more details on the proposed changes soon in hopes of getting feedback from both of you.

RDmitchell commented 4 years ago

@adrian-lara -- this issue represents the mapping presets implementation, right?

adrian-lara commented 4 years ago

Yes - that's right.

I think you may have seen and used this feature already, but just in case, we've replaced the contents of the organization column mapping setting page with a page that allows for the management of the presets. So that settings page and the actual import mapping page should be tested here.

RDmitchell commented 4 years ago

@adrian-lara -- where is the new "page that allows for the management of the presets" -- I have seen in on the mapping page, but is it also somewhere else?

adrian-lara commented 4 years ago

The "Column Mappings" tab for an organization is the page I'm talking about - a place to help users manage their presets without having to necessarily upload a file.

Screen Shot 2020-02-13 at 12 33 13 PM

RDmitchell commented 4 years ago

@adrian-lara -- ah ... I'm not sure I knew that the column mapping page had changed. I have done a bit of testing creating new mapping presets "on the fly" at the time of actually importing a file, but not here in this screen. I will do some testing there.

So if a user has a file that they haven't imported before, so there is no mapping preset for it, they can either:

import the file and create the mapping preset on the fly in the mapping screen
turn their field names into <field 1>, <field 2> and import it into the Organization / Column Mappings page and save it as a preset
In the Organization / Column Mappings page, start completely from scratch and type in each field name

right ?

adrian-lara commented 4 years ago

Yes - that's right.

RDmitchell commented 4 years ago

@adrian-lara -- found one issue when testing creating presets in Column Mappings

Instance: dev1 SHA: 6a6d4ba0 Org: LBNL Test 200

Created a new Column Mapping Preset name (Tax Lot Building List)
input field names and copied them to the Data Header column
Clicked the Copy Data File Headers directly to SEED Headers button
From the "Set All fields to" pulldown list, selected Tax Lot (even though it was the default), and the Inventory Type was NOT filled in with "Tax Lot".
I could set it by hand, but we should get the "Set all fields to:" to work as it does in the Mapping Screen

RDmitchell commented 4 years ago

However, when I left that page and then came back to it, the Tax Lot table was populated for all the Inventory Type settings. And it had made some substitutions for APN and Street Address in the SEED Header column that I had not actually made.

I did not hit the Save green checkbox button before leaving this page, so maybe we need to add the "Stay on page / Leave page" warning message saying that your settings have not been saved.

RDmitchell commented 4 years ago

@adrian-lara -- doesn't seem like this is working, unless I'm not following the correct steps

When I tried to use it when importing a file with those fields, it didn't show the mappings from the preset

Imported file: 1 - example-data-taxlots.xlsx
selected the "Tax Lot Building List" and clicked Apply Preset button
SEED HEADER and INVENTORY TYPE settings were not properly filled out
Going back to Column Mappings to make sure that the preset was properly saved, it appears to have been

RDmitchell commented 4 years ago

Tested Delete icon in Column Mappings for an existing preset, and then it was gone in the Mapping Screen, so that is working.

adrian-lara commented 4 years ago

From the "Set All fields to" pulldown list, selected Tax Lot (even though it was the default), and the Inventory Type was NOT filled in with "Tax Lot".

I could set it by hand, but we should get the "Set all fields to:" to work as it does in the Mapping Screen

I think the issue here is that after clicking "Copy into Data File Headers", the "Set all fields..." option doesn't get zero'ed out. I think it should get zero'ed out here since using that "Copy into Data..." button creates new column entries.

This thinking is coming from the following: I replicated this issue on dev1, and the "user" fix was that I just needed switch from Property back to Tax Lot. Zero'ing out that option would force a similar, but more organic, action.

adrian-lara commented 4 years ago

@adrian-lara -- doesn't seem like this is working, unless I'm not following the correct steps

When I tried to use it when importing a file with those fields, it didn't show the mappings from the preset

Imported file: 1 - example-data-taxlots.xlsx

selected the "Tax Lot Building List" and clicked Apply Preset button

SEED HEADER and INVENTORY TYPE settings were not properly filled out

Going back to Column Mappings to make sure that the preset was properly saved, it appears to have been

The Data File Headers don't match between preset and actual import file. These need to match exactly for the presets to autofill correctly. The only one that does match is City.

adrian-lara commented 4 years ago

However, when I left that page and then came back to it, the Tax Lot table was populated for all the Inventory Type settings. And it had made some substitutions for APN and Street Address in the SEED Header column that I had not actually made.

I did not hit the Save green checkbox button before leaving this page, so maybe we need to add the "Stay on page / Leave page" warning message saying that your settings have not been saved.

I could not replicate this at all, nor have I seen this before. I've tried multiple iterations of making changes, leaving the page, then revisiting the page. The changes I made that I expected to lose are never saved.

I agree with you in that I think leaving this page with changes should probably always trigger the losing data warning, vs only when you switch presets. But even when switching presets, things still seem a little buggy, so that process probably also needs some work.

RDmitchell commented 4 years ago

@adrian-lara -- you were right that I didn't choose the correct mapping preset for the file I was importing, but even when I do not all the fields are getting set properly

The Data Field headers match in the Preset and the imported file, but only 3 of them are displayed in the SEED HEADER and INVENTORY TYPE fields

adrian-lara commented 4 years ago

There's actually spaces in front of a few of the preset entries. Screen Shot 2020-02-13 at 3 44 20 PM

When importing the headers, SEED will read those spaces as part of the header.

RDmitchell commented 4 years ago

wow ... ok. I entered those as names with commas between them in the them. I don't think there were any spaces. I will test again.

RDmitchell commented 4 years ago

@adrian-lara -- making the mapping preset again. Here is my list of fields I copied them to the DATA FILE HEADERS, no spaces, saved the set, used it to import my test file, all the fields were properly filled out. So who knows how those spaces ended up in front of the fields.

Should we ignore leading spaces? I can't imagine a scenario where those would be desired ??

RDmitchell commented 4 years ago

@adrian-lara -- per your comment above "I agree with you in that I think leaving this page with changes should probably always trigger the losing data warning, vs only when you switch presets. But even when switching presets, things still seem a little buggy, so that process probably also needs some work."

RDmitchell commented 4 years ago

Instance: dev1 SHA: c97042d

See this doc -- I did a lot of testing -- works GREAT !!!

https://docs.google.com/document/d/1KK3mtBXQFkvGO-8TGsiPhfMbxifhT_fXJV524j4G3ic/edit?usp=sharing

SEED-platform / seed