Organization export is busted #299

Closed treasuretron closed 4 years ago

In GitLab by @richbodo on Dec 11, 2018, 12:10

Summary:

( Reported by @epaatdahlstrom - transcribed by @richbodo )

When logged in as admin, a GET of the URL:

https://spacebase.co/admin/exports/groups?format=csv

Should return a csv file with organization detail.

This needs to be documented here:

This used to work. It's something that spacebase admins need to do business. So it's a bug.

Why?

We use this information to mine data to upload to mailchimp or other apps. There are no end of uses to data like this for admins - all of them save a ton of time - and export beats building features to duplicate more mature external systems like mailchimp.

How about a behat test?

For bonus points - this is just a GET as admin, it should be a very simple behat test that can be based on the other working behat tests.

In GitLab by @richbodo on Dec 11, 2018, 13:11

changed title from Organzation export is busted to Organ{+i+}zation export is busted

In GitLab by @epaatdahlstrom on Dec 11, 2018, 16:04

Rich, I'm going to elevate the label for this to High since this issue is a bottle neck to an external action we've been waiting to do for sometime.

In GitLab by @richbodo on Dec 11, 2018, 22:38

marked this issue as related to #294

In GitLab by @jayelless on Dec 11, 2018, 22:58

The CSV export appears to have been removed, but is this what you are looking for: https://spacebase.co/admin/spacebase_nz_orgs

In GitLab by @epaatdahlstrom on Dec 12, 2018, 07:58

James, I don't have access to that link.

In GitLab by @epaatdahlstrom on Dec 12, 2018, 08:04

Also if it works, can somebody just run and send me the csv file for all the orgs on the website to date? Somehow I am not sure how I lost my admin access.

In GitLab by @jayelless on Dec 12, 2018, 16:04

Current export attached (170 organisations).spacebase_nz_orgs.csv

In GitLab by @jayelless on Dec 12, 2018, 16:05

The view that generates the export needs some work, as currently both the CVS export and the browser page display share the same path. Probably need to review the permissions required to access the view as well.

In GitLab by @richbodo on Dec 12, 2018, 16:06

@epaatdahlstrom You have to be logged in as admin before you visit that link.

@jayelless That's an interesting link, for sure. It renders in an odd way as html, and only shows the orgs that have owners (about 10 or 12). The full list of orgs is a little over 200. We added all of those, and as explained elsewhere, we will be contacting members of those organizations soon to encourage them to update their org profiles.

When the orgs csv worked, it used to enable us to download the entire list as a csv, including the 10 orgs that had owners and the 190 or so that do not.

The cron that creates that csv is definitely something we should talk about rebuilding. I'm sure it can be improved and documented.

The directory and the calendar are both things that we want to be able to export for analysis by business users.

In GitLab by @epaatdahlstrom on Dec 12, 2018, 16:15

Rich, yes, I sent you a slack message about the my problem with admin access.

James, there are only 57 records on the csv file even though I know all 170 orgs have email addresses and names/generic reps. Is there a way to extract all the names and emails just once to date so we can send that mass email soon? I understand there needs to be a longer term solution for this procedure. Thanks.

-Emeline

In GitLab by @jayelless on Dec 12, 2018, 16:31

@epaatdahlstrom ? When I loaded the CSV file into Excel, it showed 170 organisations. I wonder whether the file I attached was truncated?

I have made a temporary update to the live site so the CSV file can be downloaded from "https://spacebase.co/admin/export/spacebase_nz_orgs" (to avoid the path clash with the display page). It will download as a file called "spacebase_nz_orgs" which has not file type suffix, so you need to rename to add the ".csv" suffix, then it should be treated as a normal CSV file.

I will fix the view that generates the file and push the update through the normal development processes.

In GitLab by @epaatdahlstrom on Dec 12, 2018, 16:45

Thanks James. My bad. Now I can see all of it. Thanks so much!

In GitLab by @richbodo on Dec 12, 2018, 17:04

@epaatdahlstrom

You're probably the primary consumer and best to judge whether the csv is accurate, so please close this issue when QA is done. If the URL in the admin documentation is incorrect, that needs to be updated before this is closed as well.

Also, not sure what went wrong, but I didn't get a slack message from you about admin access. The admin login for spacebase.co is in lastpass and works for me. Happy to walk through logging in as admin with you or help you as much as you need elsewhere.

In GitLab by @epaatdahlstrom on Dec 12, 2018, 17:28

Understand.

Direct slack to you. I lost my access to last pass as well which is why I can’t get in as admin. Need a reset on my own account password to access Lastpass

In GitLab by @richbodo on Dec 12, 2018, 17:32

Ack! This is important! I'll call now. Spent some time searching slack for your message and it's not there, so that's an issue, too!

In GitLab by @epaatdahlstrom on Dec 12, 2018, 18:07

@jayelless , Actually I just realized that the emails for each org is not included in the csv file. That is actually the important data I need to do the mass mailings for each org. Can this be added quickly?

In GitLab by @epaatdahlstrom on Dec 12, 2018, 18:08

Separately, the people csv link does not work either so I will create another issue for that though that is not urgent right now.

In GitLab by @jayelless on Dec 12, 2018, 18:18

mentioned in commit 676c53fa0b21c03c59ad10fb21f5942526cbe864

In GitLab by @jayelless on Dec 12, 2018, 18:29

I have updated both the organizations export and the users export views to correctly export a CSV file (and to display it on the screen in batches of 100 entries, for quick review).

The links are:

Organizations: https://spacebase.co/admin/exports/organizations (screen display) and https://spacebase.co/admin/exports/organizations.csv (CSV download)
Users: https://spacebase.co/admin/exports/users (screen display) and https://spacebase.co/admin/exports/users.csv (CSV download)

In each case, the screen display links to the CSV download path for easy use.

The screen displays currently include labels for each data row, but these can be removed if required.

In GitLab by @richbodo on Dec 12, 2018, 18:35

That's awesome, James!

I'll update the urls in the admin guide.

Just got off the phone with @epaatdahlstrom , and she had some feedback on her needs for the csv files, so I'll let her chime in here.

In GitLab by @jayelless on Dec 12, 2018, 18:42

mentioned in commit 4e21a96ad328f055ebcc34afdf90f2aed9d65c3a

In GitLab by @jayelless on Dec 12, 2018, 18:43

The updates are in the branch "294-honeypot-defeated" (which is not really good practice) but this is due to be merged to dev shortly so these changes will find their way into the live system quite quickly.

They can be checked in the QA environment https://pr-246-h2h46dy-ensz4qdpjmziw.us.platform.sh/

In GitLab by @jayelless on Dec 12, 2018, 18:50

mentioned in commit a8cb971d693ec7e78e0031b678f33f15d69cd3d4

In GitLab by @jayelless on Dec 12, 2018, 18:52

mentioned in merge request !246

In GitLab by @epaatdahlstrom on Dec 12, 2018, 19:13

Not sure but somehow the new links dont work for me even when logged as admin? Error: The requested page could not be found. Am I doing something different?

In GitLab by @jayelless on Dec 12, 2018, 19:17

@epaatdahlstrom : The organization entity (ie group) does not have an email field. What email did you want added? We could add the email of the user who created the group, but as this is almost always the "admin" user, it will be of little value.

Note, the updated views are only currently available in the QA environment listed above. If they are OK I can copy to the live environment, in advance of the updates being merged as per the standard release process.

In GitLab by @epaatdahlstrom on Dec 12, 2018, 19:23

I see. It looks like my original excell file had the emails and not the orgs on the site when they were uploaded. I will have to reconstruct off line. I will use the csv file you created earlier for now as base for the 170 entries.

In GitLab by @thomasmurphy on Dec 12, 2018, 19:30

closed via merge request !246

In GitLab by @thomasmurphy on Dec 12, 2018, 19:30

mentioned in commit 2cf8b07684a0711c5af4b85e71f4afc13bcd4073

In GitLab by @thomasmurphy on Dec 12, 2018, 20:20

mentioned in merge request !247

In GitLab by @thomasmurphy on Dec 13, 2018, 00:21

mentioned in commit 4db9f691558f72d8a35c3f35590c1e35cc1c5944

In GitLab by @thomasmurphy on Dec 13, 2018, 22:48

mentioned in merge request !250

In GitLab by @thomasmurphy on Dec 16, 2018, 22:06

mentioned in commit 5812ff57c26a5a3c5ed8b08ac48d5641a11271f7

spacebase / spacebasenz

Organization export is busted #299