Search by multiple fields in the page list view

[titaniumbones]

~Admin~ users, at least, should be able to search changes by domain & URL, over particular time ranges, etc. This search function should be easy to use and responsive, and probably users should be able to bookmark particular searches so they can return to them and/or cite them once the interface becomes publicly available.

We currently have a URL-based search (well, less a search and more a pattern-match), and the API supports plenty of other filtering features that we should start with. You can see them in the API reference: https://api.monitoring.envirodatagov.org/#/pages_and_versions/get_pages

It might be best to identify individual items to add and file sub-issues for each so we can work through this incrementally. (e.g. #81 — filter by date range.)

(Updated 2019-06-17 by @Mr0grog)

[lazywithclass]

Hello @titaniumbones, could you please expand on this? I am willing to code this feature, or other if having higher priority.

I am not sure where the changes you're talking could be seen, I'm looking at my local version of the project.

[titaniumbones]

Oh jeez @lazywithclass I am a little behind on the current state of the project. The thought is:

• large numbers of web pages will make it hard for users to keep track of interesting/potentially important changes they have seen.
• It's true that "significant" changes are recorded in the database, but users will likely want to be able to search the data on other fields.
• ideally this searching can be accomplished without switching ocntexts -- so some kind of async query would make the most sense.

I think it might be more helpful to have @trinberg weigh in. @lightandluck would be the best person to discuss this, but he is away for the next couple of weeks; and @Mr0grog is badly overworked trying to pick up the slack. Toly, do you have any thoughts?

[Mr0grog]

If it helps, all the current querying features are documented on API site: https://api-staging.monitoring.envirodatagov.org/ (click on the /pages endpoint)

You can use querystrings to filter by any combination of site, agency, url (which can have asterisks for wildcards), title, source_type (e.g. only get pages that have versions from versionista or internet_archive), or hash (page has a version whose response body has this SHA-256 hash)

[Mr0grog]

And, if you get to working on this and we find other things worth filtering on, definitely file bugs on web-monitoring-db to add them!

[Mr0grog]

Hmmm, actually those docs are incomplete; you can also query on:

• capture_time - gets pages that had new versions during a given time window. Should be in the format: {start date}..{end date} where dates are in ISO 8601 format. You can also leave out either date to make the query open-ended, e.g:

  • 2017-02-01..2017-03-01 for all of February
  • 2017-02-01T10:00:00Z..2017-02-01T12:00:00 for February 1st between 10am and noon UTC
  • 2017-02-01.. for all versions from February on
  • ..2017-02-01 for all versions before February

[lazywithclass]

Thanks a lot for the explanation @Mr0grog, is @lightandluck back?

[lightandluck]

Hi @lazywithclass! Sorry, for the delay. I just got back late last week. I was roadtripping and camping the last couple weeks, so have been completely removed from the project since then. Still trying to catch up on what's happened. I'm working on a simple spec for you to help provide context for this component and should be done sometime today. Thanks for your continued interest and patience!

[lazywithclass]

No worries @lightandluck! I was also interested in understanding the priority of this feature over others, also would be great to understand how much web-monitoring-ui is used on a daily / montlhy basis.

[Mr0grog]

The UI is not actively being used, partly for lack of this feature ;)

[lightandluck]

@lazywithclass Thanks for asking those questions! In preparing, it helped me realize that this issue might not be the best place to start.

In answer to your questions, no one is using the UI right now. But the analysts, using their current workflow, have fluctuated between teams of 5-10 volunteers receiving new data sets every 3 days for the last 6+ months. I'm sure they split that work over the 3 days, so everyday, there is probably somebody analyzing data. We are steps away from pushing a v0 that will replace their current workflow, which is what web-monitoring-ui is.

In my opinion, our highest priority right now is getting a working v0 ready and encouraging quick and easy adoption as they transition to a completely new system.

We've been pushing really hard on the functional side and getting features done, but little attention has been paid to usability. In light of that, I feel the greatest contribution you could have right now is helping with improvements in user interaction which would aid in adopting the UI.

I'll open issues to explain further, as soon as I can, but as a rough guide I recommend the progression below.

1. Lurking TODOs - There are a number of TODOs in the code that haven't been properly documented or just been lurking in the back of my mind. They're minor, but good to start with and added together would greatly improve usability.

   • Indicators / Visual Cues - Currently there are no indications of processes running or loading, transitions between states of pages, etc. Everything just pops into existence or there's no feedback that an action has completed. Things we'd need: a. An indication of the record updating and when it's done. So analysts know that their changes have applied. b. Better visual indicators for significance and dictionary entries, also show saving. Right now it's just text that turns into different text. c. Transitioning between paged records. Basic data, signifiers and diffs all take different times to load and there is no indication of that. It's hard to tell what applies to which record and when.

   • Side-by-side diff analysis - One of the most used diff views is a side-by-side html render of pages that are rendered in separate iframes. The problem is that the content is variable so we can't set a correct height to provide enough space for the content. So, the user ends up with 3 windows (main app and 2 iframes) that they have to scroll all over, which is atrocious. In a very early commit of the project I was able to have the iframe set it's height onload with the snippet below, but with the switch to React, I don't know enough about the framework to get it working again. This would be very nice to have.

     iframe.onload = function() {
     // set iframe height = frame content
     iframe.setAttribute('height', iframe.contentWindow.document.body.scrollHeight);
     };

2. Current Issues - If none of the above interest you, I'd recommend first this issue #96 which is on the list for the v0 milestone. It would be a simple way to become familiar with the code and working with the API.

   After that you could dovetail neatly into this current issue. Which is fundamentally an interface for the API and filtering the lists from #96 . I don't think this current issue needs to be done for v0, but it would be the first and quickest thing we would add while transitioning. So more like for v0.1 and we would still be very glad if this is the only thing you wanted to work on. It would be merged in quickly.

If you can make it, we are having a meeting with the web dev team tomorrow (Wed) at 2-2:30pm EST. I believe it is around 7pm (London time). That would be the best way to get started, which is to talk with us. @Mr0grog and @danielballan would be there and can answer any technical questions better than I can. I'm sure one of us can stick around for awhile to talk with you. (Also, Rob and Dan please add to, clarify, or correct anything I might be wrong about.)

If you can't, just let us know what interests you the most and I'll open an issue for that task first and try to give you as much information as I can to get you started.

Also, if you can, to get a sense of our intention and why we're building this, please watch through these videos. 1.5x speed helps a lot!

• The first 30-35min is a good overview of web monitoring project as a whole.
• This is a 1 hour training video for the analysts. It's long, just skim and jump around, but It's good to see the current spreadsheet workflow that's currently used and imagine the pain points they are having.
• 15min of a recent demo
• Sneak peek

Thanks again and looking forward to your contribution!

[lazywithclass]

@lightandluck I am taking care of the TODOs right now, I am not sure I could tackle them all because I lack some context here and there, it would be great though if you could give me test credentials so I could login. I've tried with these but I think I am looking at the wrong thing.

[lightandluck]

Thanks @lazywithclass! I'm currently out all day at a workshop, but I can open up issues tomorrow to explicate.

@Mr0grog Can you set him up with a login?

[lazywithclass]

That would be great, thanks both :)

[Mr0grog]

@lazywithclass I just sent you an invite e-mail (to the e-mail on your GitHub profile) you can use to create an account on the staging server (https://api-staging.monitoring.envirodatagov.org). Just configure your .env file to have:

WEB_MONITORING_DB_URL=https://api-staging.monitoring.envirodatagov.org

…and after you’ve signed up, your login should work in the UI app.

[lazywithclass]

@Mr0grog done, it's working. Thanks!

[lightandluck]

@lazywithclass I opened up #107 and added more info to #96. I'm working on the other interaction issues I mentioned above. Don't worry about any other TODOs you see in the code, we'll get to them later.

Also, this issue is pushed to v0.1, so we'll address it then. Thanks!

[lazywithclass]

@lightandluck I've seen you've taken #96, could I do something there or what's the next highest priority issue?

[Mr0grog]

@lazywithclass I was actually about to file an issue for this, but animated SVGs don’t animate when used as a background in WebKit. Do you want to work on rewriting the <Loading> component to render the SVG inline?

(Update: filed as #118)

[lightandluck]

@lazywithclass #118 would be next highest priority. Sorry, I didn't check other browsers myself. I don't think it makes sense to split up work on #96.

We will be adding multiple issues having to deal with login, but are waiting on which permissions to add and having it implemented in -DB. It would be great if you helped tackle them. We'll ping you when that comes online and issues are created.

Let's continue any further communication on Slack, if possible. This thread is getting muddied with comments not dealing with the actual issue.

Thanks so much for your continued interest. It's really helping a lot!

[lazywithclass]

Whoops, sorry about that. I did it as a background image because I had problems understanding how to bundle the image file with the Loading component. I will try again!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in seven days if no further activity occurs. If it should not be closed, please comment! Thank you for your contributions.

[Mr0grog]

The conversation here got way off track from the original intent. Just being able to search by URL is new since this issue actually got much discussion, I think. I’d like to file some more specific features on how we should change the list/search interface and retire this.