Notify the share owner when another user downloads a shared file [$1,000]

[gig13]

Use Case:

1. User A shares a file with User B.
2. User B downloads the file via Web/desktop client/mobile.
3. User A is notified that User B has downloaded the file and via which method - Web, desktop client or mobile.

It would also be desired to have this tracked in an audit trail like the Activity or via a reporting mechanism.

@MTRichards

[LukasReschke]

Ref https://github.com/owncloud/core/issues/7812 for public link sharing

[drkmccy]

@LukasReschke Yea, for public link sharing, but more importantly for normal sharing.

This is an absolute must for a project I'm working on. Shared files with users must have traceability of who downloaded them and when. It's not so much to see if a user read an important document, but mainly to check who HASN'T read it. "Did you get the memo?" "No" "I shared it with the group" "Oh, well I didn't get it". There's an app that can sort of do it called "file counter", but it doesn't work with oc8, and theres another one called "eslog" that sends hooks to elasticsearch but it doesn't record file reads as it doesn't seem to be a filesystem hook that gets recorded.

* Willing to put a bounty on this feature *

[drkmccy]

Is there any development on this?

[bklo]

This feature is required on my project too. The FileCounter app does not suit what is needed. Any good news or should I look elsewhere?

[gig13]

@bboule @MTRichards Please confirm if this should have a milestone. Related to SF 2531

[bboule]

@MTRichards would need your guidance as to milestone? I am assuming 9.0 is full? @gig13 I am assuming this is a low severity as it seems cosmetic?

[MTRichards]

But this has traceability of who did what - the user agent string and the log app should track downloads...does it not do so?

I do not want to put this in the activity app and clutter that interface, but the log app for 9.0 should have the user agent string (the specific client and user that downloaded what). This should resolve this.

[bboule]

@MTRichards if I am reading this right, and I like to think I am, I think the ask here is if I share a file with you, I would like to be notified if you have downloaded the file and how (via my activity stream) that is to say I would like to be able to have notification (which I can turn on or off via personal page) of activity around items that I have shared with others... this seems reasonable I think what is being suggested is from the user perspective... clutter can be avoided by turning off in personal page... this would only impact the "shared"filter where I as a user can see who has accessed files I have shared....

Does this make sense?

[MTRichards]

Yep. I was keying off of this, which was only part of the comment:

Shared files with users must have traceability of who downloaded them and when

We already have this for public link sharing, which was asked above, so this is covered in activity. And we have traceability in the log files for everything else, including user agent string in 9.0 logging.

If they want notification (inside activity), this is a different user story and it has to be added to the notifications options in the personal page as another checkbox. I don't want us to confuse traceability and compliance with notification, the activity app is too easy to break and mess with to be considered true traceability for audit and compliance purposes. As long as this is fully understood, and is turned off by default, we can look at adding this hook to the activity app.

9.0 is full and well underway, could consider for 9.1 but given how fill 9.1 is, it might be tough to squeeze it in organically. Might be something that could be squeezed in around the edges since it affects the activity app (from consulting), but again I think capacity constraints are the issue there too.

[drkmccy]

To be clear. This is NOT ABOUT PUBLIC SHARING. This is tractability for local sharing. If someone has an important memo that they have shared internally with their group, that person will want to know who has and hasn't read it (downloaded it).

I have used elasticsearch to log filesystem activity but downloads are not recorded. To some, this type of activity is more important than any other type.

[MTRichards]

Are you using community edition? In that case, this is correct. Enterprise logging should be downloading and recording all of this in the ownCloud log files - just checking the state and wondering if we have a bug.

[drkmccy]

Oh I see, so download logging is an Enterprise edition only feature?

[MTRichards]

it is designed into the logs for enterprise edition, yes. The activity stream truncates itself, and a bunch of other things which can make it unreliable for logging and true traceability, thus my comments above.

[drkmccy]

So can the hook be switched on for community edition so elasticsearch can see it? Not show anything on the on the UI's activity feed, but if someone bothers to set up elastic search, they can see whats happening. The same way Elasticsearch can see a whole bunch of other data that doesnt make it to the activity feed. It'll just be one extra bit.

[bboule]

@gig13 do you have what you need based on the disposition in SF tickets? is this still needing information?

[gig13]

@bboule At this time I am all set, waiting for additional feedback from originating user

[PVince81]

The audit app should support this in 9.0, doesn't it ? @blizzz @nickvergessen

Not sure about activities though.

[nickvergessen]

Well we have the full path in admin_audit now, so that should work.

For activities we don't have the full path and neither a way to resolve it to a share owner.

But since the issue says "activity or reporting app" I think it's okay and we can close it?

[drkmccy]

@nickvergessen is this something that can be picked up by looking at non enterprise edition logs?

[blizzz]

@PVince81 well, it does not inform the owner, but the admin. If he looks into the log.

[drkmccy]

Im thinking of putting a bounty on this.

Just as a recap: ownCloud Community Edition, not Enterprise. I share an important memo with my team of 10 people. I want to be able to see who has/hasn't downloaded it.

As for interface, maybe a button can be added to the share drop down for that file, next to the trash can. pressing it can bring a popup box with everyone's name in the group and a timestamp for when it was downloaded. This would also have to work if the file was downloaded manually or automatically using the mobile app.

I think this would require the filesystem "post_read" hook put back in (it was removed in OC8).

I'll wait for replies before putting the bounty on. I also have another OC user who is interested, so he might contribute. We might be looking at a 4 figure bounty here.

[oparoz]

If I understand correctly, this is akin to the "signature folder" passing around in a corporate environment.

In terms of UI, I think it would be best to keep the info in a tab in the detailsview. You get a list of people who have been given access to the file and a check mark next to the ones who have downloaded it, along with the time stamp.

This could probably be a separate app, installable from the app store. This would give the project owner (@derekblankmccoy and co-sponsors) more freedom in terms of design and implementation.

[drkmccy]

@oparoz You got it, exactly that. Need to know that a user has read a document. I understand that is may not be technically possible, but the fact that it's been downloaded is enough. Like you say, in the details pane would be the perfect place to put it. Maybe colored arrows that match the android app: Green = downloaded latest version Yellow = downloaded but not latest

If document viewed is possible, that would be fantastic.

[oparoz]

@derekblankmccoy - Looking at the available hooks, I think we can track who has downloaded the file, but in order to be able to tell if it's been opened, then you'll need to booby-trap it. That might work with PDFs or Office files, but could be blocked by the user and is probably out of scope for this project. I don't think we have technology which can reliably tell if users have really read the document though ;).

I'm tempted to work on this if nobody else has started already, but one problem is that the sponsor's requirements are a bit different that the ones found in the OP (added notifications)

[drkmccy]

@oparoz I think your idea of having it in the details pane makes the most sense. Keep i mind that when this issue was opened, the details pane did not exist. I mean, a notification tick could be added to the notification settings, then the owner can get an email. The issue is if 50 files are shared with 50 people, that's 250 emails going out

[oparoz]

@derekblankmccoy - I hear you, but this is a blue-ticket, opened by somebody else, so unless the solution is approved by the OP's client, nobody gets paid :| . One solution is to transfer the bounty to a separate issue in which we could clearly scope the project to match the sponsor's requirements. This ticket could then add notifications, if still required. Another solution would be for @gig13 to agree with you :)

[drkmccy]

Ok, I'll wait for @gig13 to reply. If we have to transfer it, I'll write up the requirements, ready to open as a new issue.

[jospoortvliet]

It would be enough to add this to the Activity stream of the file, right? Just like it already has the ability to warn you about the download of a publicly shared file, this issue asks to have it for ALL files. Well, if it's off by default, I think it makes for a nice little feature.

That seems to be exactly what @gig13 asked for and if @derekblankmccoy is OK with that solution and puts up the bounty, he gets to decide if the solution is satisfactory anyway, no matter who opened the original ticket. I'd say - go ahead, @oparoz ;-)

[drkmccy]

That could work, but it needs to say who downloaded it with a timestamp. Also, looking at the activity tracker, how long can it get? Is there an easy way to export it to a csv?

At the moment, I'm using the esLog app and the closest I can get is seeing if a user logged in. I can only assume from there that they can see the shared file. I don't even mind if the hooks get implemented so that elasticsearch can see it, but esLog no longer works with the latest OC.

[oparoz]

That could work, but it needs to say who downloaded it with a timestamp.

That would be part of the requirements, yes.

Also, looking at the activity tracker, how long can it get?

This answer was given above: "The activity stream truncates itself, and a bunch of other things which can make it unreliable for logging and true traceability"

but I haven't looked for that limit yet. Maybe it could be lifted if the user enables the feature, but there is also this statement (emphasis mine): "the activity app is too easy to break and mess with to be considered true traceability for audit and compliance purposes."

Is there an easy way to export it to a csv?

I don't think a CSV exporter is available. That would require a script or occ command.

[drkmccy]

Ok, how about your idea of a separate app? I mean esLog would be perfect if it recorded file reads and if it worked with the latest version of OC. Maybe an app that logs file activity and exports it to a csv daily and an interface that can visualise the data.

[oparoz]

Ok, how about your idea of a separate app?

It gives you the flexibility to implement what you need, but there is something I forgot to mention. Beware that there is no guarantee that it will be maintained in a year from now unlike solutions implemented in core. Ultimately, you could end up in the same situation you are today with ESlog. Unless of course if I and others start selling support for those add-ons and people actually subscribe to get updates.

I mean esLog would be perfect if it recorded file reads and if it worked with the latest version of OC.

I took a look and there is too much "Illegal" code in that app, so it wouldn't be a good idea to patch it.

Maybe an app that logs file activity and exports it to a csv daily and an interface that can visualise the data.

That would be a different app that the one adding a "signature panel", but interesting nonetheless. Something like a table with a file on each row and users in columns, with colour coded cells? If you end up with a large table, you'll want filtering and in the end it may not be better than using search in Files and clicking on the details view icon of the files in the results to view the stats.

[oparoz]

I took a look and there is too much "Illegal" code in that app, so it wouldn't be a good idea to patch it.

Actually that may not be true. The app is outdated and should be rewritten using the AppFramework, but if what's making it incompatible with oC9.0 is outdated hooks, then fixing those could make the app generate the logs you're after.

[drkmccy]

@oparoz There is another app called "file counter" that sort of did the job, but that one doesn't work with oC8 and above.

I know there is emphasis on keeping the interface clean, so it could be kept that way. Just put the hooks in and make it log that activity in a csv.

A really cool idea would be Slack integration, that way Slack can take care of the logging.

[oparoz]

@derekblankmccoy files_counter doesn't do what you need as it only counts files downloaded from the Files app. The results would be hugely inaccurate.

I could deliver an oC9 compatible app written with the Appframework which would log to ElasticSearch the following information when a shared file is opened (to download or sync): current user, file, timestamp. I'm sure lots of ESlog features can be kept too, but I don't want this piece of work to be about trying to fix everything wrong in that app. I would let you define what you need. I would probably need to include activity notifications in order for @gig13 to mark this as solved unless we transfer the development of this app to a separate request.

There are alternatives, but you would need some solid processes in place and people who follow them.

[drkmccy]

@oparoz I mentioned ESlog because it was almost there, to be honest ElasticSearch is a bit complicated and it logs a whole load of stuff that makes no sense. Here's the list of stuff it logged:

User login User logout User creation User deletion Password change Group creation Group deletion User added/removed to/from a group Read file (it doesn't) Write file Delete file Copy/rename file Share file Enable/disablle 3rd party apps

But when Chekcing Kibana (the ElasticSearch visualizer), there's a bunch of other stuff that makes it a mess. You can eventually get to the info you're looking for, but it takes a lot of legwork. If file reads were actually there, I would have persevered with it and not put the bounty up.

Ideally, a Slack plugin for oC Monitoring would be fantastic. Is Slack integration for Activity something that you can do? The app could let you select what to log: file writes, file reads, fie deletes, file renames. We could look at possibly extending the feature to cover user logins at a future date if you wanted to do a separate piece of work (I could raise a separate issue and attach a bounty to it).

The OP mentions:

have this tracked in an audit trail like the Activity or via a reporting mechanism.

Slack would be ideal, but an app that can display it in a table that can be sorted and filtered could also work. At the very least, have it log to a csv file.

[oparoz]

@derekblankmccoy - OK, as mentioned earlier, ES requests can be tuned to send exactly what you need, but if that complicates your workflow, then there is no reason to take that path.

Ideally, a Slack plugin for oC Monitoring would be fantastic

AFAIK, that would be the same job as doing an app which connects to ES, I would just change the client. Someone could even extend the app to make it possible to pick the backend to use for logging. Since you're paying, we could stick with Slack first, but IFTTT could maybe make it more useful (if technically possible) to others as it doesn't change much in terms of privacy (you're going through 3rd party providers to log your cloud activities), but would allow admins to connect to the service they need. You could also use Mattermost, the open source Slack alternative to keep everything in-house.

Is Slack integration for Activity something that you can do?

The way I see it, that would be a different thing. You would batch process events instead of doing real-time monitoring.

Slack would be ideal, but an app that can display it in a table that can be sorted and filtered could also work. At the very least, have it log to a csv file.

Yes, there are lots of possibilities :). I'll wait for a final Statement of Work against which my work can be validated before going further. I now have a decent understanding of what can be achieved.

[drkmccy]

Ok, whichever way it goes, does oC core need to be modified at all to acknowledge filesystem reads? If so that has to be done whatever the outcome is.

[oparoz]

Ok, whichever way it goes, does oC core need to be modified at all to acknowledge filesystem reads?

For reads of files shared, no. There could be a bug in core as others are reporting issues receiving some events in their apps, but that would be a bug which needs fixing in core, not something new which needs to be approved and implemented. There is a one line patch which can be applied anyway to work around the problem and which I used for testing.

[jospoortvliet]

One thought about this - now the Activity stream might have downsides (it isn't meant to be a legal-requirements-compliant monitoring tool), but it is enough for normal human beings and there IS a API to grab the content - the desktop client shows the content right now, for example and the mobile clients will introduce this over the coming releases. So you'll have access to this trace of 'what is going on' and 'who downloaded what' from multiple places. If it can be made to do what you need, maintenance is also guaranteed in the long run - a big plus over a one-off app. Just 2 cents ;-)

[drkmccy]

@jospoortvliet The desktop client activity is no different to the activity feed on the web console. It doesn't show if a user I shared a document with has actually pressed the download button (on web or mobile client). It's very simple. When I share something with someone, I want to know if they got it or not. It's the whole point of sharing after all. I want to avoid calling several people in a group to make sure they've got what I shared with them. If I want to call anyone, I want to be able to see who didn't get it and see if they maybe have an issue logging in or whatever. Flipping it around, If someone claims they didn't get it, I should be able to see that they are telling porkies.

Scenario:There's a group of 500 students and who are already using ownCloud for course work and sharing between themselves. They are participating in an obstacle race for charity. The organizer sends directions, instructions, safety documents, and application forms through ownCloud by doing an internal share to that group of students. 400 students apply successfully. 100 claim they never got the application form. The organizer should have a way of seeing that those people did in fact download it. Let's say that out of the 400 that applied, 50 don't turn up because they say there was no directions. The organizer can then check if those people downloaded the directions. Now let's say another 10 didn't wear the appropriate clothes because the instructions didn't mention anything about it. The organizer then easily finds out that they never downloaded the instructions in the first place.

This is the sort of information that is needed. Being able to see who has and who hasn't downloaded something that was shared with them, how they downloaded it (web or mobile), and a timestamp.

[oparoz]

@derekblankmccoy - I ran through a similar scenario when trying to understand if using the Activity panel was a good idea, but I thought it would quickly be unmanageable without a lot of customisation. Let's say you get 48 notifications out of the 50 invites you've sent. How do you find the missing 2 students if you need to send them a reminder?

But if I understand correctly what @jospoortvliet is saying. Pushing notifications to Activity, as he's suggesting, would make those available to all clients (web, desktop and mobile), making it easy for event organisers to find the information on their mobile devices, without having to use the web interface or an office document app. That's only useful if the problem I mentioned above can be fixed within the Activity app and ported to all clients of course.

[drkmccy]

@oparoz So I guess the aim is to fix the activity app to include the owner's local share activity. From there, either the activity app would have to be fixed so it's got a reliable record, or we need external logging (elasticsearch, greylog, slack, whatever)

[oparoz]

@nickvergessen - What's missing in Activity to make it more robust and/or to make sure all records are kept?

[drkmccy]

The activity stream truncates itself, and a bunch of other things which can make it unreliable for logging and true traceability.

[oparoz]

Yes, but he's the author, so he may have some leads from a technical point of view. Is the limit arbitrary in order to not fill the DB or was in put in place for performance reasons. Is it time based?, etc.

There is no point in trying to fix Activity if it will never be able to meet the requirements.

[drkmccy]

Ok, so we're back to creating an app?

[oparoz]

Not necessarily. I'm hoping whatever limit is in place can easily be lifted using a setting (to be written). You will want the full log and will size your hardware accordingly if required, but a home user will probably be happy with the last 20 events or something.

The one thing I want to avoid is to log everything in Activity only to find out later that it doesn't work for you.

[oparoz]

Custom app

• Can log to the DB (as good or bad as Activity)
• Can log to an external service (SLA determined by 3rd party)
• Could offer an API
• Best effort maintenance without subscription
• No need to get any sort of green light by the core team
• Should not require any hacking of core
• May not meet the OP's requirements

Activity app

• Currently truncating logs
• Deemed unreliable by core Product Manager
• Logs to the DB (may have scaling issues if adding more events?)
• Offers the information via an API
• API is used by official clients to get the list of events
• Officially maintained

[nickvergessen]

Well we have a background job that prunes after 365days by default. Value can be changed via config: https://github.com/owncloud/activity/blob/master/lib/backgroundjob/expireactivities.php#L64-L66

The otherthing is downloading via webdav (syncing with client or federated share) does currently not trigger any hook, so they are not detectable.