search

Xunius / Menotexport

Python solution to export annotations from your Mendeley library.
GNU General Public License v3.0
123 stars 20 forks source link

Database Encryption #25

Open matteosecli opened 6 years ago

matteosecli commented 6 years ago

Hello, I'm having troubles with the database after the last Mendeley update. My current situation is

They are both synced to each other; however, Menotexport can read the database on the Linux box but not on my MacOS laptop. The error I get is the following:

# <Menotexport>: Failed to recoganize the given database file.
file is not a database

I'm pretty sure the problem is due to the fact that Mendeley has recently decided to encrypt the user database "to comply with GDPR & keep your data safe [...]" (which I think it's totally bull**it & it's not what GDPR says). I've found a comment on the issue & some instructions to obtain a decrypted database here, but I was not able to test them on my updated Mendeley install on MacOS because it seems that the MacOS version doesn't have a debug mode.

I'm reporting this as a bug even if it's actually Mendeley's fault; however, I think it could be useful to keep track of the issue.

Thanks again for your work! 🙂

Xunius commented 6 years ago

Hi matteosecli,

Many thanks for the report! It's a big thing, if I can't figure out a way to deal with the encryption then this tool will be dead. I'll put a note in the readme.

williamgunn commented 6 years ago

Hey @Xunius, we're really sorry we broke your tool with the encryption! We want to get it working again and provide a way to access the local database that we can officially support so that tools will keep working as we make changes. We're currently collecting use cases and setting up some one-on-ones with the development team. If you're interested, please email me.

maphouse commented 5 years ago

has this been solved? As a mendeley user (version 1.19.2), can i still use this tool? Would appreciate a definite answer before cloning. Thanks @Xunius !

tuxflo commented 5 years ago

I'm also interested since in version 1.19.3 there is no option to export highlights at all (only sticky notes). I just thought I found a great tool to do the research work for my thesis...

matteosecli commented 5 years ago

@maphouse @tuxflo I think you can still download and use either Mendeley 1.18 (Stable) or Mendeley 1.18.1 (Development) as a workaround. I suggest to install the old version on another machine or on a virtual machine in order to avoid potential conflicts with the newer version.

Here are the links (directly from Mendeley's servers) of these old versions:

Mendeley 1.18 (Stable)

Mendeley 1.18.1 (Development)

tuxflo commented 5 years ago

Thanks for the quick reply. I just installed the Windows Version 1.18.1 but it seems that there is no export feature as well.... maybe I'd just look for some alternative to Mendeley although I really like the Annotation & Sync feature... and it seems to be the only "integrated" tool that runs on Linux as well.

matteosecli commented 5 years ago

@tuxflo have you tried to open the PDFs one-by-one and then select "File -> Export PDF with Annotations"? I have Mendeley 1.19.2 and this feature is still present, though Mendeley does not have bulk-exports and the annotations are exported in a non-standard way -- that's also why Menotexport was made.

Anyway I fully understand your point: it's one of the best reference managers out there and I also need a multi-platform solution, too bad they keep "closing" it more and more.

tuxflo commented 5 years ago

You are totally right: it is possible to do that when the document is open in the internal PDF Viewer (I already edited my comment but for some reason it wasn't saved)

Currently I'm facing troubles with installing your tool: do I have to install some python dependencies first? And do I have to clone the repo or is conda doing that as well (sorry I never heard of conda before...)

matteosecli commented 5 years ago

@tuxflo this tool was written by @Xunius (not by myself), so it's actually better to wait for his answer for this sort of stuff! 😃

tuxflo commented 5 years ago

OK Nevermind, I just skipped the conda stuff and did a regular install and now everything is just fine! I'm using 18.1, it syncs and the (editable) bulk annotation works. Thanks so much for your help @matteosecli and @Xunius for creating this awesome little tool :+1:

matteosecli commented 5 years ago

Glad that it works! 😄

Xunius commented 5 years ago

@tuxflo Glad that this little thing helps.

@matteosecli Thanks for helping out. I was away the previous week. And I just noticed that the PDFs we are exporting don't preserve metadata (some pieces of info the original PDF creator inserts into the file, like title, author, and even doi). For ordinary user this won't be noticeable at all, but for others it may be a pity to lose them as it's much easier to recognize a document from its metadata than parsing the file. So I'm planning to add this. Also I'm "secretly" making a rip off a mendeley (working name "elemendy" XD). Of cause it won't be as powerful but it will be totally transparent. I would let you know if you are interested, and when it gets somewhere (currently it's still preprepre alpha).

tuxflo commented 5 years ago

@Xunius Yes I'm interested! I just had the same idea (build a open source alternative to mendeley) yesterday. Currently I don't have time to do some coding myself (writing my thesis, thats why I came in touch with mendeley) but here are just some thoughts I had:

If you are still working on the project after march next year I'd be happy to help to get this thing done :) I have some knowledge in different programming languages, especially in C++ and Qt (but python is good as well)

matteosecli commented 5 years ago

Hi @Xunius, sorry for the very long absence! I thought I would have had less work after October, but at this point I think it was just a sci-fi dream and the work kept increasing and increasing...😅

Anyway I really like your idea! I was trying to look into that by myself a couple of years ago, but I really don't know anything about user interfaces (I'm a command-line guy) so I quickly gave up. But I'd be very interested if you let me know when you have something, at least I can help you testing or (if I have time) give some contributions to the backend part.

About the metadata problem...do you already have some hint about the cause and (possibly) the solution? Unfortunately I don't know so much about the PDF library you're using here. Otherwise one could always resort to a brute-force thing like: copy the meta before adding annotations and then paste them back.

Xunius commented 5 years ago

Hi @tuxflo @matteosecli Glad to hear positive feedbacks from both of you (I just realized that we probably have a Mendeley guy in this thread, but that's ok I guess).

I can totally appreciate that we all have our own priorities, my own progress has been slowed down a lot recently.

I agree with tuxflo that I'm not going to implement any syncing and will let the user use whatever to their liking. Auto-save is on my plan as well. In terms of pdf viewer, I have little idea how difficult it would be to embed a viewer into this, my plan (at least for the 1st stage) is to let the user fire up an external viewer, and move onto an embedded one should the 1st stage goes smoothly.

A few other things on my list:

That' about it, still a whole lot to be done. If you get any idea please let me know. Winter holiday is coming so I should be able to make some process soon.

sojusnik commented 5 years ago

Last year, a Mendeley update went live that had the unintentional consequence of hindering interoperability. We made a change to the Mendeley Desktop application that broke some integrations with users’ local Mendeley database, including Zotero’s import tool, which resulted in it not working with Mendeley. We’re really sorry about this — it was never our intention to break these integrations and we should have picked this up in release testing.

We’ve heard a lot from our users about this and our team has been working behind the scenes on two things.

First, we clearly heard from users that they want to be more in control of how and when they can export their PDFs, annotations and highlights directly from Mendeley. Improvements were released in Mendeley Desktop version 1.19.3 and further improvements to the export process will be released as part of Mendeley Desktop version 1.19.4. source

Let's see what the future will bring.

Xunius commented 5 years ago

Hi @matteosecli @tuxflo,

It has been long time since we talked about the "rip off" project. I haven't been idle, it took me much longer than I expected to come up with the 1st alpha. But here it is finally (https://github.com/Xunius/MeiTingTrunk).

As I mentioned previously, the database is a sqlite, with a structure similar to Mendeley's but simplified. I've managed to implement import/export of bibtex and RIS. DOI query is added, and autosave is also there. But the big thing missing is citation style formatting, and full text searching in PDFs is lacking atm. I also want to implement import from Zotero, but I guess I need to make this more appealing first, otherwise Zotero users won't be attracted in the first place.

I haven't got time to pack it up in an easier-to-install format (I assume that won't be too difficult), so currently one has to install the dependencies manually, But those are pretty standard packages and can be install easily with pip.

Please give it a look if you are still interested. Btw, have you tried the newer versions of Mendeley, did they remove the encryption?

matteosecli commented 5 years ago

Hi @Xunius, thanks a lot for the update! It already looks like a great work! 😄

I'll test it more thoroughly and come back here asap! 😉

(btw, newer versions of Mendeley still seem to have encryption onboard)

JoTrab commented 4 years ago

I found a solution to circumvent the encryption problem. After a lot of researching around and always coming back to this page I stumbled over the following: https://eighty-twenty.org/2018/06/13/mendeley-encrypted-db

The author explains a procedure to de-encrypt the sqlite. I am right now testing a direct zotero import from the de-encrypted file as well as a menotexport extraction on the de-encrypted file. So far it works like a charm.

I had to run the x/s $rdi procedure twice due the $1 = 8 instead of $1 = 0 error as suggested by the author. If you guys could link/add to the manual or if you feel include the procedure to the program I guess a lot of people will be quite happy. At least I can finally move on to zotero with my very large literature database. Keep up the good work

Xunius commented 4 years ago

@JoTrab Sorry I totally forgot about your comments. I've added your suggested solution to README. Many thanks.