ToDo List

[jarun]

Continued from #233.

Notes

The list below is a growing one. While suggesting new features please consider contributing to Buku. The code is intentionally kept simple and easy to understand with comments. We'll be happy to assist any new contributor. We need your help!

Some of the fresh-baked features may not have been released yet. Grab the master branch for those.

Identified tasks

• [ ] Paging option (10 by default if value is omitted in option). The current behaviour stays the same as users already have scripts based on it.
• [ ] Android app (with the same schema) (probably a separate project; we'll be glad to help)
• [x] Show results before deletion
• [x] Support keyword filtering (records having keywords a and b but not c and d) (thanks @SaltyCatFish)
• [x] Support filtering by tags with search options (see #250) (thanks @SaltyCatFish)
• [x] Port feature from googler/ddgr - omniprompt key O
• [x] Show results with most search keyword matches on top (thanks @mosegontar)
• [x] Text-mode user agent for Buku
• [X] Copy search result URL to cipboard
• [x] Read default Firefox profile name from profiles.ini (see #212, thanks @alex-bender)
• [x] Support --format in search results (ref, thanks @mosegontar)
• [x] API documentation (comments need to be in NumPy format) (thanks @mosegontar)
• [x] Auto-import: optionally add parent folder name as tag, ask for unique tag [like importdb()]
• [x] Support custom colours (thanks @shv-q3)
• [x] Generate packages on Travis-CI using PackageCore (see #189) (thanks @shaggytwodope)
• [x] Search multiple tags, exclusion in tag search (thanks @mosegontar)
• [x] Auto-import Firefox and Google Chrome bookmarks (thanks @alex-bender)
• [x] Suggest tags those go together
• [x] Append/overwrite/remove tags from prompt
• [x] Rest API for webapps (thanks @kishore-narendran)
• [x] Add more tests (ongoing activity @rachmadaniHaryono)
• [x] A browser plugin (thanks @SamHH for bukubrow)
• [x] Text editor support (thanks @ZwodahS)
• [x] Need a PyPI maintainer (thanks @shaggytwodope)
• [x] Make refreshdb faster using threads (record updates should be synchronized)
• [x] Show usage count in tag list
• [x] Proxy support (thanks @denisfalqueto)
• [x] Continuous search at prompt
• [x] Add prompt help
• [x] Specify custom DB file to class BukuDb (library usage, no exposed option)
• [x] Move to urllib3
• [x] Handle redirects using referrer masking. Example URL. Fixed with urllib3.
• [x] Support URL shortening. This helps to share URLs. (see #92 for limitations)
• [x] Make a bookmark title immutable via refreshdb()
• [x] Markdown import/export
• [x] Regex search
• [x] Ubuntu PPA (thanks @shaggytwodope)
• [x] Export specific tags to HTML
• [x] Exact word match using REGEX. Make substring match optional.
• [x] Delete all records based on a search result
• [x] Delete multiple items, support combination of indices and ranges
• [x] Append tags
• [x] Travis CI integration
• [x] Ubuntu deb package generation on new tag
• [x] Merge bookmark database files (for users who work on multiple systems)
• [x] Export bookmarks in FF or Chrome html format.
• [x] Option to add folder names as tags while importing HTML (thanks @mohammadKhalifa)
• [x] Check and show upstream version
• [ ] Anything else which would add value (please discuss in this thread)

[rachmadaniHaryono]

related to idea on #255

example: https://github.com/rachmadaniHaryono/Buku/tree/feature/server

screenshots

currently only for view function on bookmark and tags. the style is taken from bootstrap starter template https://getbootstrap.com/docs/3.3/examples/starter-template/

there could be more advanced feature such whitelist/blacklist tag, so there will be + or - sign after the tag to do that

[tom-i]

Looks pretty awesome, @rachmadaniHaryono are you going to integrate function for having buku-server on some other network place (server) than localhost?

[jarun]

Awesome!!! We must have it in the main project :). Probably we can add a directory named buku_server and have the implementation there. Slowly you can add the functionality we have in core. I am pathetic with GUI stuff but I believe we will get contributors as we keep adding ToDo items.

[rachmadaniHaryono]

i haven't tried with other but you can put it on 0.0.0.0 so other device can reach it

https://stackoverflow.com/questions/31904761/what-does-app-runhost-0-0-0-0-mean-in-flask

so run the command like this

bukuserver run --host 0.0.0.0

see also more run option with bukuserver run --help

e: command is changed from buku-api to bukuserver

[rachmadaniHaryono]

Probably we can add a directory named buku_server and have the implementation there

imo we can rename the api folder to buku_api or buku_server and change the setup for the new path. the name api is too common and may conflict with other package

e:

I am pathetic with GUI stuff but I believe we will get contributors as we keep adding ToDo items.

i am also learning and most of this is done with less html and without javascript for now.

e2: changed both folder and command to bukuserver

[jarun]

buku_server would be fine.

Where do I run the command buku-api run --host 0.0.0.0 in your tree?

[rachmadaniHaryono]

first just install it in your virtualenv

pip install -e .
pip install -e .[server]

the run the command

[jarun]

buku_server would be fine.

I meant we can rename api to buku_server.

the run the command

cool!

[rachmadaniHaryono]

should the command be changed as well (from buku-api to buku-server)?

[jarun]

How about bukuserver in both cases?

[rachmadaniHaryono]

done, will edit recent comment to match new command

[tom-i]

@rachmadaniHaryono What am I doing wrong? :-(

(env) $ pip install -e .[server]
Obtaining file:///home/tomik/programs/buku/Buku
  buku 3.6 does not provide the extra 'server'
Installing collected packages: buku
  Found existing installation: buku 3.6
    Uninstalling buku-3.6:
      Successfully uninstalled buku-3.6
  Running setup.py develop for buku
Successfully installed buku

(env) $ bukuserver run --host 0.0.0.0
bash: bukuserver: command not found

EDIT: Ok after some missing modules in virtualenv I finally can run:

./server.py run

Unfortunatelly there is 404 on page:

(env) [api] $ ./server.py run
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [23/Mar/2018 23:21:07] "GET / HTTP/1.1" 404 -
127.0.0.1 - - [23/Mar/2018 23:21:07] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [23/Mar/2018 23:21:18] "GET /bookmarks HTTP/1.1" 404 -
127.0.0.1 - - [23/Mar/2018 23:21:19] "GET /favicon.ico HTTP/1.1" 404 -

[rachmadaniHaryono]

@tom-i

• first make you are on this commit 250758b1aaf8c5a131bdba84416f552590d3407a (run git show)

• check if setup.py contain entrypoint like below

  entry_points={
      'console_scripts': ['buku=buku:main', 'bukuserver=bukuserver.server:cli']
  },

  • check also if setup.py have extra server dependencies

    extras_require={
    'HTTP': ['urllib3'],
    'CRYPTO': ['cryptography'],
    'HTML': ['beautifulsoup4'],
    'tests': tests_require,
    'server': server_require,
    },

• check if templates file exist on bukuserver/templates/bukuserver folder. there should be 4 html files

• i also make it possible to run it directly on server.py like so python bukuserver/server.py but i don't recommend it, use bukuserver command instead.

also this is optional, but report if favicon is still showing error

[jarun]

Sorry, Friday night is family movie time and I was off the terminal! :) We watched Paddington 2.

I guess we need to document all the steps under a README.md under bukuserver directory and link to it from the main README.md. Not urgent, but eventually.

[rachmadaniHaryono]

For documentation I can only think of 2 points

• additional installation package. See more below
• running the server. Show how to run on user defined host & port.

I just realize that on ubuntu installation it is recommended to install from ppa. I haven't check if required server package is on ppa so I recommend only to install from pip for now

---

Also maybe inform user that this feature is still on experimental stage

[jarun]

:+1:

[tom-i]

Hi @rachmadaniHaryono, I've looked to that git show and I had this one.. so not yours :-/ I don't know, how is it possible

commit 474c403521f89798952ac1c5e4b7c09546411487 (HEAD -> master, origin/master, origin/HEAD)
Author: rachmadani haryono <rachmadaniHaryono@users.noreply.github.com>
Date:   Fri Mar 23 11:30:13 2018 +0800

but git pull shows me: Already up to date

also entry_points have not that last part about 'bukuserver=bukuserver.server:cli' and extras_require there is missing 'server': server_require,

So it looks, that I'm using wrong commit or so :-/ Because I have there api folder and not bukuserver as you have it in your git repository :(

Buku [ ls -l                                                                                                                               master * ] 10:55 AM
total 296
-rw-r--r-- 1 tomik tomik  14461 Mar 23 22:04 CHANGELOG
-rw-r--r-- 1 tomik tomik  35142 Mar 23 22:04 LICENSE
-rw-r--r-- 1 tomik tomik    125 Mar 23 22:04 MANIFEST.in
-rw-r--r-- 1 tomik tomik    508 Mar 23 22:04 Makefile
-rw-r--r-- 1 tomik tomik  22988 Mar 23 22:04 README.md
drwxr-xr-x 2 tomik tomik   4096 Mar 23 22:33 __pycache__
drwxr-xr-x 4 tomik tomik   4096 Mar 23 23:12 api
drwxr-xr-x 5 tomik tomik   4096 Mar 23 22:04 auto-completion
-rw-r--r-- 1 tomik tomik  24529 Mar 23 22:04 buku.1
drwxr-xr-x 2 tomik tomik   4096 Mar 23 22:22 buku.egg-info
-rwxr-xr-x 1 tomik tomik 145022 Mar 23 22:04 buku.py
drwxr-xr-x 3 tomik tomik   4096 Mar 23 22:04 docs
drwxr-xr-x 5 tomik tomik   4096 Mar 23 23:05 env
-rw-r--r-- 1 tomik tomik   1531 Mar 23 22:04 packagecore.yaml
-rw-r--r-- 1 tomik tomik     58 Mar 23 22:04 requirements.txt
-rwxr-xr-x 1 tomik tomik   1877 Mar 23 22:22 setup.py
drwxr-xr-x 2 tomik tomik   4096 Mar 23 22:04 tests
-rw-r--r-- 1 tomik tomik   1042 Mar 23 22:04 tox.ini

EDIT: Aha.. I'm using @jarun git repository :-/ But it's strange I thought that I've used @rachmadaniHaryono :-/ I'll try to repair it..

[rachmadaniHaryono]

Try this to get server branch https://stackoverflow.com/questions/9153598/how-do-i-fetch-a-branch-on-someone-elses-fork-on-github

E: you can also add other fork to your local git repo. See git remote add

[tom-i]

buku/Buku [ git remote add rachmadaniHaryono https://github.com/rachmadaniHaryono/Buku.git
buku/Buku [ git fetch rachmadaniHaryono                                                                                                                 master ] 11:05 AM
From https://github.com/rachmadaniHaryono/Buku
 * [new branch]      feature/codecov                    -> rachmadaniHaryono/feature/codecov
 * [new branch]      feature/exception-on-malformed-url -> rachmadaniHaryono/feature/exception-on-malformed-url
 * [new branch]      feature/fix-idx                    -> rachmadaniHaryono/feature/fix-idx
 * [new branch]      feature/fix-server                 -> rachmadaniHaryono/feature/fix-server
 * [new branch]      feature/release-branch             -> rachmadaniHaryono/feature/release-branch
 * [new branch]      feature/server                     -> rachmadaniHaryono/feature/server
 * [new branch]      master                             -> rachmadaniHaryono/master

buku/Buku [ git checkout -b rachmadaniHaryono/Buku                                                                                                      master ] 11:07 AM
Switched to a new branch 'rachmadaniHaryono/Buku'

buku/Buku [ git pull rachmadaniHaryono/Buku                                                                                             rachmadaniHaryono/Buku ] 11:08 AM
fatal: 'rachmadaniHaryono/Buku' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

What should I put to pull?

[rachmadaniHaryono]

You can checkout the branch git checkout rachmadaniHaryono/feature/server

[tom-i]

@rachmadaniHaryono I'm noob with this, anyway how to pull your branch?

buku/Buku [ git pull                                                    rachmadaniHaryono/feature/server ] 6:11 PM
There is no tracking information for the current branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.

    git pull <remote> <branch>

If you wish to set tracking information for this branch you can do so with:

    git branch --set-upstream-to=<remote>/<branch> rachmadaniHaryono/feature/server

I'm using yours, but git pull or git pull rachmadaniHaryono/feature/server doesn't do anything :-/

[jarun]

Try

git remote add upstream ...

Otherwise, wait till we merge this on Buku master branch and then you can test it out much easily.

[tom-i]

Hmm doesn't help. How long that merge will take, please? A week?

[jarun]

We don't know at this moment. Please have patience.

[rachmadaniHaryono]

git remote add rachmadaniHaryono https://github.com/rachmadaniHaryono/Buku.git
git checkout rachmadaniHaryono
git checkout -b rh-server rachmadaniHaryono/feature/server

That will create rh-server branch on you local repo

Currently I don't have my laptop until few hours later, so please check the command again

[tom-i]

@rachmadaniHaryono ha.. cool that helps ;) Now I see that page with my bookmarks ;) Tomorrow, I'll try to install it on my EspressoBin server, to check if it's working with --host 0.0.0.0 ;) Thx man..

[rachmadaniHaryono]

note on doc conversion for pypi upload

2 example from last time (#233) is already fixed directly by pypandoc

now it seems example 1 to 9 is not formatted correctly

see https://test.pypi.org/project/buku/3.7/

pypandoc seems merge some line on one of those example

other than that removing raw html part should make generated rst file ready to upload

[jarun]

Fine by me. Not a major issue.

[RyanRLong]

@jarun FYI, not sure if anyone else is working on it yet, but I plan on having a PR for paging this weekend.

[jarun]

@SaltyCatFish that would be awesome!

[tsaoyu]

Any interests on orgmode markdown .org export? So that bookmarks can be integrated with GTD.

[jarun]

So that bookmarks can be integrated with GTD.

which service is GTD?

[tsaoyu]

@jarun GTD, stand for get things done, here it is not a service but a way to manage bookmarks within personal workflow. Suppose we come across GitHub.com and we want to read it in the future. It is easy to do so with orgmode by exporting .org list from buku. See example here

[jarun]

OK, I have an idea now. How about importing a .org file? Also, would you raise a PR for this?

[jarun]

@SaltyCatFish can you please let us know how you are planning to implement paging? We can have a short review of the design and all of us can be on the same page as to what we expect. Please raise an issue for the discussion and add the details so we can review it.

[RyanRLong]

Will do.

On Fri, Mar 30, 2018, 9:37 AM Arun Prakash Jana notifications@github.com wrote:

@SaltyCatFish https://github.com/SaltyCatFish can you please let us know how you are planning to implement paging? We can have a short review of the design and all of us can be on the same page as to what we expect. Please raise an issue for the discussion and add the details so we can review it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jarun/Buku/issues/251#issuecomment-377525749, or mute the thread https://github.com/notifications/unsubscribe-auth/AF5IH9GQZRt9b-6PeIhmrSuI6NsFZxWpks5tjjUYgaJpZM4St7lU .

[rachmadaniHaryono]

• http://blog.jonparrott.com/github-flavored-markdown-on-pypi/
• https://dustingram.com/articles/2018/03/16/markdown-descriptions-on-pypi

This means release branch can be deleted and no conversion needed for next release

[jarun]

👍

[jarun]

@SaltyCatFish any update on the paging task?

[RyanRLong]

Sorry, not yet.

[jarun]

No problem! I know it's tricky. You can get some clues from ddgr interface and code.

Thanks for the update!

[RyanRLong]

Awesome, thanks!

[jarun]

@SaltyCatFish any update on paging?

[RyanRLong]

No, I've bit off a bit more than I can chew with some other projects. I still hope to spend some quality time with the code base this week, but if anyone else wants to tackle this request, feel free.

[jarun]

if anyone else wants to tackle this request, feel free

I don't think it's happening... even I have one pending task and I can't find time due to other things... but please try to spare some time on it.

[RyanRLong]

@jarun How should this work? Does the user type "buku -paging-option 10 20" to see rows 10 - 20, or do you want a something like unixs less/more?

[jarun]

The results should be paged:

buku -n 10 ...

should show 10 results per page and one should be able to navigate with the n and p keys at the prompt. Just run googler and ddgr for a feel of it.

[rachmadaniHaryono]

https://github.com/rachmadaniHaryono/Buku/tree/feature/statistic

no link yet to statistic page and user have to go directly e.g. 127.0.0.1:5000/statistic

the chart is using chartjs from http://chartjs.org/docs/latest/

it can more expanded, e.g.

• tag statistic
• most common word on description,
• domain to tag relationship
• nlt for description

FUN idea: Badge:

• 0, 10, 100, 1k bookmark/tag
• sfw/nsfw/category bookmark achieved (have x website bookmark with max or min number)
• unique domain award (when unique domain is the most dominant)
• website bookmark (only bookmark certain website or it is 80% of the bookmark)

---

here is kinda concept that i imagine for additional feature on bukuserver. user/developer/thirdparty can add plugin which can be shown on each page.

this is kinda tricky because

• how plugin is managed
• enable/disable plugin
• plugin location
• link to plugin
• html/js/css?
• position on main page

[jarun]

We do not track usage data. If we want to have this feature (and it looks great) we should have it disabled by default and enable only if user wants it.

plugin

there are issues around the DB desing as well shared DB: pugin may corrupt it separate DB: plugin may get out of sync

[rachmadaniHaryono]

We do not track usage data. If we want to have this feature (and it looks great) we should have it disabled by default and enable only if user wants it.

Not quite sure if this is categorized as tracking usage data, because all data calculated each time per request and the result is not saved anywhere. Also no other database request other the first time getting all records. So Imo opinion it is similar to display number of your bookmarks

Of course unless the flask bootstrap or the included chart.js is compromised, it will be different story

should have it disabled by default and enable only if user wants it.

There is several way I can think of to do this

• soft confirmation : just yes/no confirmation either the first time or every time user enter statistic page
• environment variables : user have to set BUKUSERVER_STATISTIC to enable the page

dB

I have no idea how to approach this. I imagine that the new side database never hold any buku index and buku dB is always prioritized (except some exceptional case). The disadvantage is more database request

The side database can hold url, tag, title and produce/process something about it.

E.g.Url icon plugin (or something similar). first server get requested url after that looking for paired icon in side database

---

But still this is just some idea because I think it is already out of scope of what buku should be.

If people really want it may have to create another repo for this