search

finaldie / auto-news

A personal news aggregator to pull information from multi-sources + LLM (ChatGPT/Gemini/Ollama via LangChain) to help us reading efficiently with less noises, the sources including: Tweets, RSS, YouTube, Web Articles, Reddit, and personal Journal notes.
MIT License
519 stars 56 forks source link

How to use ? #95

Open SachaSecours opened 3 months ago

SachaSecours commented 3 months ago

Hello, I just installed it on my computer. Now I wonder how to use this tool? How to select my different sources? How to watch the news? ...

Thanks

finaldie commented 3 months ago

hi @SachaSecours

Glad to know you have finished the installation process. For the reading part, we use Notion as the UI:

  1. From the setup wiki, assume you have created your Notion entry page, then open your Notion entry page
  2. Now it's time to add your favorite RSS feed/Reddit channel in this step: Go to Index page, then open RSS_List and Reddit_List to add and enable them
  3. Last part, Open ToRead/ToRead page, this is the Notion database as your main information hub, all the posts will feed into it, and you can create different View/Filters based on your preferences, it's up to you to set up a special view. You may want to check this video to have some initial ideas. For myself, I created 9+ views to help me focus on the different category I care about)

image

As a quick example, click the + button to create a new view on the database, we can choose different style (default is Table), then from the View options, we can assign the View name, add filters, sort by columns, etc.

image

After the first View creation, you may want to tweak it a little, create more Views for your needs, add/delete RSS_List and Reddit_List to collect more posts. There is a great Notion database views tutorials put as a reference.

Also, I highly recommend installing the Notion Chrome extension. You can add your favorite Web Post or YouTube video to your Inboxes directly, and the auto-news backend will help you process it automatically.

Enjoy the journey with auto-news, and let me know if any questions.

SachaSecours commented 3 months ago

Hello, I have a lot of questions,

Thanks

finaldie commented 2 months ago

@SachaSecours

If no Notion token and main page created, auto-news backend won't do anything, this is the first step we need to do: https://github.com/finaldie/auto-news/wiki/Docker-Installation

SachaSecours commented 2 months ago

Thanks for the answers.

Should the integration be public? If yes, what should I put in the fields? Because after running all the commands, the Notion page still looks like this :

Capture d’écran 2024-09-02 à 15 16 05

And I checked, the integration is well connected to the project with all the permissions, and that's why I wonder if it should be public or private. For now it is private.

finaldie commented 2 months ago

@SachaSecours

Great! yes, keep it private, and all the posts will only be available to you. (Unless you want to share with others, you can later decide whether to open to the public or other individuals)

SachaSecours commented 2 months ago

And I also have these error :

! milvus-ui The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s ! airflow-triggerer The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s ! airflow-scheduler The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s ! airflow-webserver The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s ! airflow-worker The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s ✔ Container docker-airflow-init-user-1

And airflow-init-1 and airflow-init-user-1 are not running :

https://github.com/user-attachments/assets/1ce3988e-7e14-4e99-8e82-68b2690d32bd

finaldie commented 2 months ago

airflow-init-1 and airflow-init-user-1 are normal; those jobs are designed to be run once and exit; (If you want to double-check details, click on those jobs to verify the logs, there shouldn't be any errors if everything is correct)

SachaSecours commented 2 months ago

Ok but why it's not working on Notion ?

SachaSecours commented 2 months ago

I made a summary of everything I did, can you tell me if it is correct and if I did not make a mistake:

Thank you very much for your help, I hope I was precise enough

finaldie commented 2 months ago

@SachaSecours

Thank you for listing the details. I feel that it's really close to making it work for you.

As the notion sub-pages cannot be created, I'd want to make sure the NOTION_ENTRY_PAGE_ID is correct, do you mind to have a check, this var contains: 6c....a6 or Auto-news-Page-6c....a6?

As we only need the ID part, which we need to set with the below example in the build/.env:

NOTION_ENTRY_PAGE_ID=6c....a6

Let me know which one you are using.

The other one

"OAuth client secret key" corresponds to "NOTION_TOKEN" in the Makefile

I suppose you want to mention you have set the NOTION_TOKEN (from the Notion Integration page) into the build/.env, not the Makefile.

For example, from your Notion integration item, copy that Internal Integration Secret, and use it to set the NOTION_TOKEN in build/.env

image

SachaSecours commented 2 months ago

Thanks,

So I changed my NOTION_ENTRY_PAGE_ID from Auto-news-Page-6c....a6 to 6c....a6so NOTION_ENTRY_PAGE_ID=6c....a6.

I suppose you want to mention you have set the NOTION_TOKEN (from the Notion Integration page) into the build/.env, not the Makefile.

Yes you're right.

And I use the good secret for the variable NOTION_TOKEN in build/.env

But the NOTION page still empty.

I tried to restart it with :

make stop
make deps && make deploy
make start

Let me know if I missed a step or made a mistake somewhere. Thanks again for your answer

finaldie commented 2 months ago

Thanks for the confirmation, please use below command to start from a clean stage (remove the workspace folder)

make stop
rm -rf workspace
make deps && make deploy && make start
SachaSecours commented 2 months ago

Yes, it's working !

Now, how to use it on Notion, how to configure it ?

Thanks

finaldie commented 2 months ago

@SachaSecours ,

Congrats! Please follow the wiki to set up and configure your Notion, add the RSS list and Reddit channels you want, and then wait for the new posts to come in.

Feel free to have a try, and let me know if you have any questions.

Config RSS and Reddit list

It's time to add your favorite RSS feed/Reddit channel in this step: Go to Index page, then open RSS_List and Reddit_List to add and enable them

Configure Notion database views

Open ToRead/ToRead page, this is the Notion database as your main information hub, all the posts will feed into it, and you can create different View/Filters based on your preferences, it's up to you to set up a special view. You may want to check this video to have some initial ideas. For myself, I created 9+ views to help me focus on the different category I care about)

image

As a quick example, click the + button to create a new view on the database, we can choose different style (default is Table), then from the View options, we can assign the View name, add filters, sort by columns, etc.

image

After the first View creation, you may want to tweak it a little, create more Views for your needs, add/delete RSS_List and Reddit_List to collect more posts. There is a great Notion database views tutorials put as a reference.

Notion browser extension (Nice to have)

Also, I highly recommend installing the Notion Chrome extension. You can add your favorite Web Post or YouTube video to your Inboxes directly, and the auto-news backend will help you process it automatically.

Enjoy the journey with auto-news, and let me know if any questions.

aurlro commented 2 months ago

@SachaSecours can you tell me if it's work for you ?

YanWuHao commented 2 months ago

hi, I got same issue, after I execute "make start" I got an empty notion page,I read all issue and I don't know how to solve this problem. is there any log for this problem ? below is everything I did:

  1. create a notion page
  2. create a notion token
  3. check the token is working
  4. edit .env file and copy it to build/.env
  5. execute "make deps && make deploy && make start" by the way ,the notion page id makes me confuse, according to notion doc we should insert hyphens into share string, does we need insert hyphens into NOTION_ENTRY_PAGE_ID filed in .env file? ( I tried all but my notion page still empty, so sad) notion_page_id notion doc Sorry I'm not good at English
YanWuHao commented 2 months ago

I think I found the problem , it seems like docker compose --env-file not work? my computer is mac studio

docker compose -p autonews --env-file ../build/.env up  airflow-init-user
WARN[0000] The "AIRFLOW_UID" variable is not set. Defaulting to a blank string.
[+] Running 10/5
 ✔ Network autonews_bot_network                                                                                                                                     Created0.0s                            0.1s
 ✔ Container autonews-redis-1                                                                                                                                       Created0.1s                            0.1s
 ✔ Container autonews-postgres-1                                                                                                                                    Created0.1s
 ✔ Container mysql-db                                                                                                                                               Created0.1s
 ✔ Container autonews-airflow-init-1                                                                                                                                Created0.1s
 ! airflow-init The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested      0.0s
 ✔ Container autonews-airflow-scheduler-1                                                                                                                           Created0.0s
 ! airflow-scheduler The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
 ✔ Container autonews-airflow-init-user-1                                                                                                                           Created0.0s
 ! airflow-init-user The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 0.0s
Attaching to airflow-init-user-1

airflow-init-user-1  | User: uid=50000(airflow) gid=0(root) groups=0(root)
airflow-init-user-1  | PWD: /opt/airflow
airflow-init-user-1  | ----
airflow-init-user-1  | Apply patches ...
airflow-init-user-1  | /home/airflow/.local/lib/python3.11/site-packages/langchain/__init__.py:29: UserWarning: Importing LLMChain from langchain root module is no longer supported. Please use langchain.chains.LLMChain instead.
airflow-init-user-1  |   warnings.warn(
airflow-init-user-1  | /home/airflow/.local/lib/python3.11/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
airflow-init-user-1  |
airflow-init-user-1  | `from langchain_community.chat_models import ChatOpenAI`.
airflow-init-user-1  |
airflow-init-user-1  | To install langchain-community run `pip install -U langchain-community`.
airflow-init-user-1  |   warnings.warn(
airflow-init-user-1  | 1 Database initializing...
airflow-init-user-1  | MySQL client initialization finished, host: None, port: None, user: None
airflow-init-user-1  | Traceback (most recent call last):
airflow-init-user-1  |   File "/home/airflow/airflow/run/auto-news/src/patches.py", line 63, in <module>
airflow-init-user-1  |     apply_patches()
airflow-init-user-1  |   File "/home/airflow/airflow/run/auto-news/src/patches.py", line 35, in apply_patches
airflow-init-user-1  |     db_cli.init_tables()
airflow-init-user-1  |   File "/opt/airflow/run/auto-news/src/mysql_cli.py", line 34, in init_tables
airflow-init-user-1  |     self._create_table_patch()
airflow-init-user-1  |   File "/opt/airflow/run/auto-news/src/mysql_cli.py", line 37, in _create_table_patch
airflow-init-user-1  |     conn = self.connect()
airflow-init-user-1  |            ^^^^^^^^^^^^^^
airflow-init-user-1  |   File "/opt/airflow/run/auto-news/src/mysql_cli.py", line 26, in connect
airflow-init-user-1  |     return mysql.connector.connect(
airflow-init-user-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^
airflow-init-user-1  |   File "/home/airflow/.local/lib/python3.11/site-packages/mysql/connector/pooling.py", line 322, in connect
airflow-init-user-1  |     return CMySQLConnection(*args, **kwargs)
airflow-init-user-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
airflow-init-user-1  |   File "/home/airflow/.local/lib/python3.11/site-packages/mysql/connector/connection_cext.py", line 140, in __init__
airflow-init-user-1  |     self.connect(**kwargs)
airflow-init-user-1  |   File "/home/airflow/.local/lib/python3.11/site-packages/mysql/connector/abstracts.py", line 1360, in connect
airflow-init-user-1  |     self.config(**kwargs)
airflow-init-user-1  |   File "/home/airflow/.local/lib/python3.11/site-packages/mysql/connector/abstracts.py", line 667, in config
airflow-init-user-1  |     self._port = int(config["port"])
airflow-init-user-1  |                  ^^^^^^^^^^^^^^^^^^^
airflow-init-user-1  | TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'
airflow-init-user-1  | Apply patches finished
airflow-init-user-1  | ----
airflow-init-user-1  | Enable DAGs...
airflow-init-user-1  | Dag: news_pulling, paused: False
airflow-init-user-1  | Dag: sync_dist, paused: False
airflow-init-user-1  | Dag: collection_weekly, paused: False
airflow-init-user-1  | Dag: journal_daily, paused: False
airflow-init-user-1  | Dag: action, paused: False
^CGracefully stopping... (press Ctrl+C again to force)
[+] Stopping 1/1
 ✔ Container autonews-airflow-init-user-1  Stopped
YanWuHao commented 2 months ago

I changed the docker-compose.yaml ,it works . Sorry I'm not good at docker env

airflow-init-user:
    <<: *airflow-common
    entrypoint: /bin/bash
    # yamllint disable rule:line-length
    command:
      - -c
      - |
        echo "User: `id`"
        echo "PWD: `pwd`"
        echo "----"
        echo "Apply patches ..."
        python3 ~/airflow/run/auto-news/src/patches.py
        echo "Apply patches finished"
        echo "----"
        sleep 5
        echo "Enable DAGs..."
        airflow dags unpause news_pulling
        airflow dags unpause sync_dist
        airflow dags unpause collection_weekly
        airflow dags unpause journal_daily
        airflow dags unpause action
        # airflow dags unpause upgrade
        echo "Enable DAGs finished"
    env_file: '../build/.env'
    environment:
      <<: *airflow-common-env
    user: "${AIRFLOW_UID:-50000}:0"
    depends_on:
      airflow-init:
        condition: service_completed_successfully
      airflow-scheduler:
        condition: service_healthy
      mysql:
        condition: service_healthy
finaldie commented 2 months ago

@YanWuHao , glad you've figured that out, I believe you are not using the latest code, as the latest code has addressed this issue which will copy the build/.env into runtime https://github.com/finaldie/auto-news/blob/main/docker/docker-compose.yaml#L336

Feel free to open a new issue if it doesn't work for you, cheers!

SachaSecours commented 2 months ago

@SachaSecours can you tell me if it's work for you ?

Yes, everything is good for me !

SachaSecours commented 2 months ago

@finaldie, do you know if I can host it on a server ? Which one ?

finaldie commented 2 months ago

@SachaSecours

Yes, you can host it in any server you prefer, AWS, GCP, or your personal server, etc

Do it as usual, checkout the code in server, and copy the build/.env there, then use the same instruction as you deploy in your local machine via docker.

Additionally, if you want to deploy the backend via k8s, e.g. Helm or ArgoCD, check the following wikis will help: