developkariyer / iwapim

A compact ERP solution with lots of connectors to e-commerce systems. Based on Pimcore
https://developkariyer.github.io/iwapim/
Apache License 2.0
5 stars 0 forks source link

Automated Listing & Update Listing #57

Open enesbt opened 4 weeks ago

enesbt commented 4 weeks ago

We believe we are ready to push product feeds to marketplaces. I'm currently conducting tests to enable listing through both a user interface and the console. Since each marketplace operates differently, we need different solutions, database designs, and interfaces tailored to each one. The most critical issue is category and product attributes. These attributes are regularly updated by marketplaces, and we need to utilize this data accurately. By enabling AI to select these attributes, we can fully automate the listing process. Using Ollama, we can run popular AI models locally on our server (e.g., gemma3, qwq, deepseek-r1, llama3.2 ...). "https://dev.to/robertobutti/consuming-http-streams-in-php-with-symfony-http-client-and-ollama-api-p9g" Since it is not actively used and is easier compared to other marketplaces, we could start with Çiçeksepeti. Based on the results we achieve here, I believe we can then proceed to other marketplaces.

developkariyer commented 3 weeks ago

I tried some AI but prompts should be crafted carefully and also the data feeded with the prompt should be normalized for LLM to return an acceptable result. Anyway, directly using Gemini-AI in free tier might resolve context window limitation.

enesbt commented 2 weeks ago

Objective

To enable any user to list selected products on preferred marketplaces with a single click — quickly, efficiently, and without errors.

Process Steps

  1. Product and Variant Selection The user selects desired products and variants from the catalog via the system interface.
  2. Marketplace Selection The user determines which marketplaces the selected products will be listed on (e.g., Amazon, Trendyol, Hepsiburada, etc.).
  3. Queueing Tasks Marketplace-specific tasks are created for each selected product. These tasks are written into the queue systems assigned to each marketplace (Symfony Messenger).
  4. Data Gathering and Preparation Each marketplace uses its own handler module to process queued tasks. Missing product data is completed, formatting is adjusted, and the data is prepared to comply with the marketplace’s API requirements.
  5. Product Listing Prepared product data is sent via the respective marketplace APIs, and the products are listed on those platforms.
  6. Logging and Audit Trail All processing steps, including successes and failures, are thoroughly logged and recorded by the system.
  7. User Feedback & Reporting Once the process is completed, users receive a summary report detailing the actions taken for each product on each selected marketplace.

🚧 Challenges

  1. Queue Infrastructure Setup Each marketplace should have its own queue system to support independent and parallel processing. The infrastructure must be scalable and dynamic, allowing new marketplaces to be added in the future without requiring major architectural changes.
  2. Missing Data & AI Integration Each marketplace has different requirements for product information. To fill in missing data using AI, well-crafted, context-aware prompts tailored to each marketplace must be created to ensure accurate content generation.
  3. Category & Attribute Management Marketplaces have unique category structures and attribute systems. A dedicated queue system should handle synchronization and updates. AI can assist in mapping categories and attributes, but again, high-quality prompts will be essential for accuracy.
  4. Product Pricing Strategy Since product prices often vary per marketplace, each platform requires its own pricing logic. AI can assist with dynamic pricing, but liability for potential pricing errors lies with us, not the AI. Cost-based pricing strategies using product costs, marketplace commissions, and shipping fees are currently managed via Excel. A future recommendation is to transition this logic into a centralized system via Pimcore.
enesbt commented 1 week ago

02.05.2025 Progress Report Notes

General Progress

The queue structure has been successfully implemented with marketplace distinction. No issues occurred during the process.

The Gemini integration has been successfully completed.

AI Observations

Despite temperature=0, the AI sometimes generates fabricated responses.

It cannot determine product prices.

It can select the correct category for a new product based on previously used categories.

It cannot directly select attribute ID values (due to input size limitations).

It can extract attribute values like color and size (useful for rule-based systems).

It can select images to be uploaded from provided sample listings.

Attribute Management

For the category chosen by the AI, rule-based SQL queries can be executed to fetch ID values based on color and size.

This is valid in cases where product variants depend on color and size.

However, when different marketplaces enforce required attribute fields, rule-based systems may become insufficient.

While color and size values are standard across most marketplaces, our catalog includes many non-standard values, which prevents rule-based systems from functioning properly.

Auto Listing Suggestions

To develop a fully automated system:

Infrastructure must be built for all marketplaces.

Categories, attributes, and price planning should be predefined for selected product groups. However, I don't believe this would offer a significant advantage over manual listing.

Before implementing auto-listing, the following steps should be taken:

Standardize the catalog.

Standardize marketplace listing structures.

Calculate product costs.

Store product images in the catalog for each variant.

Conclusion

For now, bulk listings can be done for predefined products based on current needs. (Ciceksepeti)

However, transforming this into a fully automated system does not seem feasible at this stage.

Bonus - ÇiçekSepeti API Integration Updates

A sample listing has been uploaded to ÇiçekSepeti via API.

Although more than 48 hours have passed, the listing is still in the "processing" state. (Manual uploads are typically processed within 24 hours.)

If the test listing via API is successful:

Both success and error events will be saved in the database.

API responses will be temporarily stored in the tmp folder.

This will enable batch operations to be executed in an organized manner.