search

open4good / open4goods

The open4goods project
Other
16 stars 10 forks source link

readme

The project

Beta Production

An open source online comparator that operates ecological scoring, following a common good mindset.

Technical metrics and maven site is deployed on Github Pages:

Open4goods (o4g) is an open-source and open-data product aggregator, search engine and comparator. More over, it is a stack aiming at handling large product datasets identified by GTIN's. It is build upon Maven, Java, SpringBoot, Elasticsearch, Redis, and some other cool libraries. It is mainly designed to :

How to contribute

There are several ways to contribute to the project.

Use it

By buying your products on official sites, you "create" money through the affiliation system, that allows to reverse the "by law" 20% environmental compensation. By using it you will also able to provide us some feedbacks

Speak about it

Communication will be the lack of this Odyssey, any kind of help is welcome. Talking about this the project to your granny or even to your dog could help us a lot.

Feedback us

Bugs and ideas are greatly welcome ! We have a specific feedback system that allows you to report feedback to Github Issues directly from your navigation on our frontends.

Eco-score and vertical definitions

Verticals (eg : tv's, washing machines, ...) are defined through yaml configuration files. The ecoscore is part of the verticals configurations.

Our verticals definitions are totaly open, meaning that you are very welcome to :

Geek it !

Contribute to the websites, on the UI, on the content, or on the data aspects. Quiet simple to set-up, you can run open4goods website localy with the below instructions.

Run in dev mode

We will see here how to run open4goods frontends and API's on tour computer.

Software Requirements

You will need :

running through docker-compose

Elasticsearch and Redis are used by the open4goods project. A Kibana instance is commented in the docker-compose.yml file, if you want to browse data's by yourself. Go to the project root, then start the compose file

docker compose up

Docker compose is managed by spring-boot-docker-compose. That means that the manual launching of docker-compose is not mandatory, containers will be started by the application if not present.

Elastic, kibana and Redis should be available on :

Before running the docker compose check that the value of 'vm.max_map_count' is higher or equal to 262144. If not, you will have to raise your max map args to be able to rune the Elastic image, see the Hint's section

Building the open4goods project from code base

Jars are not published to any central repo (nor planned to, it seems not to make any particular sense). To build, please go into the project folder. Then :

mvn install

This will build and run tests, hope in your terminal you'll get a

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for parent 0.0.1-SNAPSHOT:
[INFO]
[INFO] parent ............................................. SUCCESS [ 2.509 s]
[INFO] commons ............................................ SUCCESS [ 27.046 s]
[INFO] crawler ............................................ SUCCESS [ 8.990 s]
[INFO] verticals .......................................... SUCCESS [ 7.143 s]
[INFO] api ................................................ SUCCESS [ 11.289 s]
[INFO] ui ................................................. SUCCESS [ 7.895 s]
[INFO] test ............................................... SUCCESS [ 1.496 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:07 min
[INFO] Finished at: 2021-01-31T15:20:30+01:00
[INFO] ------------------------------------------------------------------------

Launching

The open4goods project is packaged under the form of several SpringBoot web applications. You will probably want to launch :

Command line

From the project root folder, you can now launch the aspects of the platform you are interested in :

UI

java -Dspring.profiles.active=dev -jar ui/target/ui-[VERSION].jar

You should be able to access the open4goods user interface at http://localhost:8082

Api

java -Dspring.profiles.active=dev -jar api/target/api-[VERSION].jar

You should be able to access the open4goods API at http://localhost:8081

Crawler

You should not need to run a separate crawler, since an embedded one is instanciated in the API. However, if you want to play, or register as a open4goods web scrapper and help us in crawling the world, you could setup an individual crawler node.

java -Dspring.profiles.active=dev -jar target/bin/open4goods-crawler.jar

You should be able to access the open4goods crawler interface at http://localhost:8080

Using an IDE

If using any kind of IDE (tested with Eclipse and Intellij), please import as maven or SpringBoot project, then run or debug from the following Application classes :

Don't forget to specify the profile to use (probably you'll want "dev" to run in development mode)

Loading some datas

For now, you have an up and running open4good platform. You will like to play with some datasets, and like it's not so easy to get them, we provide some live datas directly from our websites.

Products auto-loading

A special service will automaticaly load sample datas on applications startup, allowing you to easily play with the project.

A note on the work directory

TODO : Document

A note on Xwiki

TODO : Document

How to's

TODO : Document

Play with the UI

TODO : Document

Play with the data

TODO : Document

Ingest new data

TODO : Document

Transform data : the aggregation mechanism

Hints

Report issues from code

TODO That means that simple PR's to TODO can directly make new products of enhance the user experience of all open4goods users.

Elastic max map count

You will probably need to raise your max map args in order to be able to start elastic

sudo  sysctl -w vm.max_map_count=262144

To permanently set the max virtual memory :

  1. edit /etc/sysctl.conf
  2. add line vm.max_map_count=262144
  3. sudo service sysctl restart

Elastic space crash

After a space crash, elastic indexes are locked. You can get hand back on them by using :

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'