Implement download reports

[andrewvanbreda]

Need download reports for:

User: own data they have entered Co-ordinators: for their region Super-users: whole dataset

Possibly clone functionality from PoMS/eBMS for this

[andrewvanbreda]

Hi @DavidRoy

Can I clarify that I am doing the right thing before I spend the time on it.

One possible plan is to implement the following:

1. Add a download option to the existing "My fixed square results" map and grid page.

2. Create a new page "My transect walk results" with similar features as above.

3. Add a new regional results, complete download page which would use Indicia easy_download form. I think this can be done on one page with role detection built in.

Shout if you want something different to the above.

Questions before going ahead.

Currently EU Poms doesn't have the concept of someone having a region (the countries don't work in the same way as EBMS) This could work in a similar way as FIT Count though, and appear as a selectable drop-down on their account page? I assume the regions are the same as the countries on the data entry pages?

[DavidRoy]

Yes, that's right and the countries are the same as for the data entry pages

[DavidRoy]

@andrewvanbreda this is the next priorty for SPRING. Do you have an eta for having this ready to test?

[andrewvanbreda]

@DavidRoy Yes am in middle of this. Think the transect bit is mostly ready to test, but I made a major change today that hasn't been pulled yet.

If you need urgently I can probably get you a testable version in next day or so.

[DavidRoy]

That's good to know. In the next few weeks is fine, not super urgent

[andrewvanbreda]

Hi @DavidRoy,

These are now ready for testing (I have tested them myself also).

A few notes

1. The pages to check are these.

https://pollinator-monitoring.net/my-transect-walk-results https://pollinator-monitoring.net/transect-walk-results https://pollinator-monitoring.net/fixed-square-results

Note these are not currently available in the menu yet, and are only available if you have administrator rights (which you do). That will be change when they go live and normal users will have access to the My Data report.

All the pages have separate sample and occurrences grids.

2. The pages use a report explorer as this can then display a map. We could use the same reports on Easy Download if you prefer. Use the Download option in bottom right of the grids to download.

Note that the downloaded CSV files contain extra columns to the grids seen on screen. Fewer columns are shown on screen purely to stop the grid being too wide. The columns displayed are just chosen by me (i.e. a guess), so if you prefer others, then let me know.

3. The "Fixed square results" and "Transect walk results" pages support use of the Regional Coordinator role. There is a drop-down on the Drupal account page to set the region.

Administrators should not have the region set if you want them to access all the data.

Note that regional coordinators who have not had their region set will also see all the data, so their region should be set.

4. I am aware of a problem where one of the users is not having their Sun % and Cloud % fields filled in, so there is no need to report that issue to me. This data appears to be missing from the cache table, am not sure why yet.

5. There is a "My fixed squares results" page already live in the menu, so that is why that isn't in the list of new reports to test.

Andy

[DavidRoy]

Thanks. Looking good. A few immediate comments.

1. The https://pollinator-monitoring.net/fixed-square-results second table is lacking the plant names?
2. The occurrences grids do not need to include the recorder name as this comes in the sample download?

Can you also make available for 'Regional co-ordinators' so I can ask them to test

[andrewvanbreda]

Hi @DavidRoy Have done the Regional Coordinator part of this (I also checked one of the coordinators can actually access the pages after masquerading). You will need to set the "Coordination country" drop-down on the corrdinators' account page.

Not done the other bits yet.

[andrewvanbreda]

@DavidRoy Those problems have been fixed.

[DavidRoy]

@andrewvanbreda could you add these pages to the 'View results' pages, reflecting permissions settings

[andrewvanbreda]

@DavidRoy This is deployed now. I will close this after I check the glitch in the cache table I mentioned above.

[andrewvanbreda]

Hi @BirenRathod Could you send me the results of these 3 queries please. Probably best to email it to me rather than post here as not sure is suitable for Github. Thanks

select * from indicia.samples where id in ( 20286744, 20286712, 20061851, 20061829, 20061753, 20061803, 20286863, 20286855, 20286805, 20286794, 20286682, 20286673, 20286667);

select * from indicia.cache_samples_functional where id in ( 20286744, 20286712, 20061851, 20061829, 20061753, 20061803, 20286863, 20286855, 20286805, 20286794, 20286682, 20286673, 20286667);

select * from indicia.cache_samples_nonfunctional where id in ( 20286744, 20286712, 20061851, 20061829, 20061753, 20061803, 20286863, 20286855, 20286805, 20286794, 20286682, 20286673, 20286667);

[BirenRathod]

@andrewvanbreda Here are the results. spring.zip

[andrewvanbreda]

Thanks @BirenRathod

@johnvanbreda I have a problem with this project, the attrs_json field is not filled in sometimes, but I don't undeerstand why. Isn't this an automatic process? Have you seen that happen before? Example sample number is 20061753 which appears to have sample attributes the the attrs_json is empty.

[johnvanbreda]

Coincidentally I had noticed that this ties in with the spatial indexing not working this morning, so I think that sometimes the work queue doesn't complete all the tasks it should. I can't see why but I have added some logging code to the warehouse so I'll have a look tomorrow to see whether there are new samples without spatial indexing or attrs_json. If so I should be able to get a bit more info.

[johnvanbreda]

I've changed the way the work_queue works so it won't cleanup any tasks that haven't completed properly - not 100% sure this will fix it but hopefully it will. I've also queued the blank ones so they will get filled in correctly.

[andrewvanbreda]

Closing, as data looks like it is now filled in, and the reports have past all other tests and are live. If there are an individual data caching problems they can be raise separately.