search

metanorma / stepmod-utils

Tools for working on the STEPmod repository.
1 stars 0 forks source link

Copy referenced images from resource docs #185

Closed HassanAkbar closed 1 year ago

HassanAkbar commented 1 year ago

Will copy the referenced image to resource folder,

closes #184

Intelligent2013 commented 1 year ago

@HassanAkbar I didn't use stepmod-utils before. Could you point me how to use it?

I've tried:

git clone https://github.com/metanorma/stepmod-utils
cd stepmod-utils
git checkout copy_referenced_images_from_resource_docs
bundle install

The command

bundle exec ./exe/stepmod-convert-express-resource D:\Work\Metanorma\repositories\iso-10303-stepmod\data\resource_docs\draughting

outputs

bundler: command not found: ./exe/stepmod-convert-express-resource
Install missing gem executables with `bundle install`

Then I've tried:

cd exe
bundle exec stepmod-convert-express-resource D:\Work\Metanorma\repositories\iso-10303-stepmod\data\resource_docs\draughting\resource.xml

The output is:

D:/Work/Metanorma/repositories/stepmod-utils/lib/stepmod/utils/converters/figure.rb:12:in `pattern': [figure]: no state given, 2 (StandardError)
        from D:/Work/Metanorma/repositories/stepmod-utils/lib/stepmod/utils/converters/figure.rb:27:in `convert'
        from C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/reverse_adoc-0.3.7/lib/reverse_adoc/converters/base.rb:11:in `treat'
        from C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/reverse_adoc-0.3.7/lib/reverse_adoc/converters/base.rb:6:in `block in treat_children'
        from C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/nokogiri-1.14.5-x64-mingw-ucrt/lib/nokogiri/xml/node_set.rb:235:in `block in each'
        from C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/nokogiri-1.14.5-x64-mingw-ucrt/lib/nokogiri/xml/node_set.rb:234:in `upto'
        from C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/nokogiri-1.14.5-x64-mingw-ucrt/lib/nokogiri/xml/node_set.rb:234:in `each'
        from C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/reverse_adoc-0.3.7/lib/reverse_adoc/converters/base.rb:5:in `inject'
        from C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/reverse_adoc-0.3.7/lib/reverse_adoc/converters/base.rb:5:in `treat_children'
        from D:/Work/Metanorma/repositories/stepmod-utils/lib/stepmod/utils/converters/fund_cons.rb:13:in `convert'
ronaldtse commented 1 year ago

@Intelligent2013 you should use the gem from its released form, ie. “gem install stepmod-utils”

ronaldtse commented 1 year ago

@HassanAkbar can you please help address the issues that @Intelligent2013 has encountered? We need to demo this feature tomorrow. Thanks.

HassanAkbar commented 1 year ago

@Intelligent2013 This will copy the images when generating the annotated files and for that you need to do the following steps:

cd <project directory>
rake install
stepmod-annotate-all <path to stepmod data>
Intelligent2013 commented 1 year ago

@HassanAkbar thank you for the answer, but I can't find the referenced image in the resource folder. Current folder: D:\Work\Metanorma\repositories\stepmod-utils\ git branch:

* copy_referenced_images_from_resource_docs
rake install
stepmod-annotate-all D:\Work\Metanorma\repositories\iso-10303-stepmod\

The folder D:\Work\Metanorma\repositories\iso-10303-stepmod\data\resources\draughting_element_schema contains:

descriptions.xml
developer.xml
draughting_element_schema.exp
draughting_element_schema.xml
draughting_element_schemaexpg1.gif
draughting_element_schemaexpg1.svg
draughting_element_schemaexpg1.xml
draughting_element_schemaexpg2.gif
draughting_element_schemaexpg2.svg
draughting_element_schemaexpg2.xml
draughting_element_schemaexpg3.gif
draughting_element_schemaexpg3.svg
draughting_element_schemaexpg3.xml
draughting_element_schema_annotated.exp
resource_map.xml

The file draughting_element_schema_annotated.exp has the reference to the image annotaion_to_model_leader_line_circle_as_provided.gif:

(*"draughting_element_schema.des_apll_point_symbol.circle.__figure"
[[figure-draughting_element_schema-7]]
====
image::annotaion_to_model_leader_line_circle_as_provided.gif[]
====
*)

but annotaion_to_model_leader_line_circle_as_provided.gif is missing in D:\Work\Metanorma\repositories\iso-10303-stepmod\data\resources\draughting_element_schema.

It presents in the source folder D:\Work\Metanorma\repositories\iso-10303-stepmod\data\resource_docs\draughting\annotaion_to_model_leader_line_circle_as_provided.gif

What am I doing wrong?

HassanAkbar commented 1 year ago

@Intelligent2013 It is working on my system I think it has something to do with the folder path for windows. I have updated the code to cater to that but I was unable to confirm it because I don't have a windows system.

Can you try the following commands:

cd <project directory>
git pull origin copy_referenced_images_from_resource_docs
rake install
stepmod-annotate-all <path to stepmod data>
Screenshot 2023-10-18 at 7 38 53 PM
Intelligent2013 commented 1 year ago

@HassanAkbar the same issue, no .gif.

Intelligent2013 commented 1 year ago

@HassanAkbar I've found the issue.

In stepmod_file_annotator.rb:

      def resource_docs_schemas(stepmod_dir)
        filepath = File.join(stepmod_dir, "data", "resource_docs", "*", "resource.xml")

filepath is D:\Work\Metanorma\repositories\iso-10303-stepmod_test_utils\data/resource_docs/*/resource.xml (path with mixed / and \ separators) and Dir.glob(filepath).each ... doesn't return anything, therefore resource_docs_cache is empty.

My command line was:

stepmod-annotate-all D:\Work\Metanorma\repositories\iso-10303-stepmod_test_utils\

I've replaced \ to /:

stepmod-annotate-all D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils/

and now it's working, the folder D:\Work\Metanorma\repositories\iso-10303-stepmod_test_utils\data\resources\draughting_element_schema\ contains all necessary .gifs.

I.e. as I understand, stepmod_dir is user's defined path (File.join isn't using here), therefore we need somewhere replace \ to / in the stepmod_dir string.

Intelligent2013 commented 1 year ago

https://stackoverflow.com/questions/7173000/slash-and-backslash-in-ruby

ronaldtse commented 1 year ago

In this case the directory path should first be instantiated as “Pathname.new” for the path to be reasonable.

HassanAkbar commented 1 year ago

@Intelligent2013 I've pushed the code to handle paths with mixed slashes so it should work on windows now.

cc: @ronaldtse

Intelligent2013 commented 1 year ago

@HassanAkbar confirmed, now .gif's present in the resource/<schema name>/ folder.

But sometimes there are error messages on the console log: [ERROR]!!! conflicting chdir during another chdir block I don't remember that I see them previously.

Queueing next batch
Queueing next batch
4300: Queued processing D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils//data/resources/aic_csg/aic_csg.exp
4320: Queued processing D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils//data/resources/action_and_model_relationships_schema/action_and_model_relationships_schema.exp
[ERROR]!!! conflicting chdir during another chdir block
D:/Work/Metanorma/repositories/stepmod-utils/lib/stepmod/utils/stepmod_file_annotator.rb:349:in `chdir'
D:/Work/Metanorma/repositories/stepmod-utils/lib/stepmod/utils/stepmod_file_annotator.rb:349:in `convert_from_resource_file'
D:/Work/Metanorma/repositories/stepmod-utils/lib/stepmod/utils/stepmod_file_annotator.rb:72:in `block in call'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/nokogiri-1.14.5-x64-mingw-ucrt/lib/nokogiri/xml/node_set.rb:235:in `block in each'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/nokogiri-1.14.5-x64-mingw-ucrt/lib/nokogiri/xml/node_set.rb:234:in `upto'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/nokogiri-1.14.5-x64-mingw-ucrt/lib/nokogiri/xml/node_set.rb:234:in `each'
D:/Work/Metanorma/repositories/stepmod-utils/lib/stepmod/utils/stepmod_file_annotator.rb:65:in `call'
D:/Work/Metanorma/repositories/stepmod-utils/exe/stepmod-annotate-all:54:in `block (3 levels) in <top (required)>'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'
4320: Queued processing D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils//data/resources/action_schema/action_schema.exp
4300: Done processing aic_csg.exp => D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils//data/resources/aic_csg/aic_csg_annotated.exp.
4300: Queued processing D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils//data/resources/aic_curve_swept_solid/aic_curve_swept_solid.exp
[ERROR]!!! conflicting chdir during another chdir block
D:/Work/Metanorma/repositories/stepmod-utils/lib/stepmod/utils/stepmod_file_annotator.rb:349:in `chdir'
D:/Work/Metanorma/repositories/stepmod-utils/lib/stepmod/utils/stepmod_file_annotator.rb:349:in `convert_from_resource_file'
D:/Work/Metanorma/repositories/stepmod-utils/lib/stepmod/utils/stepmod_file_annotator.rb:72:in `block in call'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/nokogiri-1.14.5-x64-mingw-ucrt/lib/nokogiri/xml/node_set.rb:235:in `block in each'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/nokogiri-1.14.5-x64-mingw-ucrt/lib/nokogiri/xml/node_set.rb:234:in `upto'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/nokogiri-1.14.5-x64-mingw-ucrt/lib/nokogiri/xml/node_set.rb:234:in `each'
D:/Work/Metanorma/repositories/stepmod-utils/lib/stepmod/utils/stepmod_file_annotator.rb:65:in `call'
D:/Work/Metanorma/repositories/stepmod-utils/exe/stepmod-annotate-all:54:in `block (3 levels) in <top (required)>'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
C:/tools/ruby31/lib/ruby/gems/3.1.0/gems/concurrent-ruby-1.2.2/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'
4320: Queued processing D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils//data/resources/aic_advanced_brep/aic_advanced_brep.exp
4300: Done processing aic_curve_swept_solid.exp => D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils//data/resources/aic_curve_swept_solid/aic_curve_swept_solid_annotated.exp.
4300: Queued processing D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils//data/resources/aic_draughting_annotation/aic_draughting_annotation.exp
4320: Done processing aic_advanced_brep.exp => D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils//data/resources/aic_advanced_brep/aic_advanced_brep_annotated.exp.
4320: Queued processing D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils//data/resources/aic_associative_draughting_elements/aic_associative_draughting_elements.exp
HassanAkbar commented 1 year ago

@Intelligent2013 

4300: Queued processing D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils//data/resources/aic_csg/aic_csg.exp
4320: Queued processing D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils//data/resources/action_and_model_relationships_schema/action_and_model_relationships_schema.exp

By looking at the thread ids (4300, 4320) in the logs you posted, it looks like it is running in multiple threads. This might have been caused if you changed it locally to run using multiple threads here -> https://github.com/metanorma/stepmod-utils/blob/main/exe/stepmod-annotate-all#L32

This is expected behaviour for chdir in ruby if running in multiple threads as mentioned here -> https://bugs.ruby-lang.org/issues/15661 so we updated the code to run in single thread mode in this issue -> https://github.com/metanorma/stepmod-utils/issues/139.

Intelligent2013 commented 1 year ago

This might have been caused if you changed it locally to run using multiple threads here -> https://github.com/metanorma/stepmod-utils/blob/main/exe/stepmod-annotate-all#L32

@HassanAkbar Very interesting, I didn't change it... Current value:

MAX_THREADS = 1 #[2, Concurrent.processor_count].max * 2

And the same value in D:\Work\Metanorma\repositories\stepmod-utils\pkg\stepmod-utils-0.3.30.gem -> data.tar.gz -> data.tar -> exe/stepmod-annotate-all.

HassanAkbar commented 1 year ago

@Intelligent2013 This is interesting this might have to do something with ruby 3

@ronaldtse if we are only using a single thread then should we remove the pool structure and only use a single loop? This will prevent issues like this on all ruby versions.

ronaldtse commented 1 year ago

@Intelligent2013 This is interesting this might have to do something with ruby 3

@ronaldtse if we are only using a single thread then should we remove the pool structure and only use a single loop? This will prevent issues like this on all ruby versions.

I’m fine to keep it single threaded.

HassanAkbar commented 1 year ago

@ronaldtse I've removed the support for multithreading from stepmod-annotate-all.

Queueing next batch
Queueing next batch
4300: Queued processing D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils//data/resources/aic_csg/aic_csg.exp
4320: Queued processing D:/Work/Metanorma/repositories/iso-10303-stepmod_test_utils//data/resources/action_and_model_relationships_schema/action_and_model_relationships_schema.exp
[ERROR]!!! conflicting chdir during another chdir block

@Intelligent2013 this should be fixed now.

Intelligent2013 commented 1 year ago

@Intelligent2013 this should be fixed now.

@HassanAkbar confirmed, thank you!

ronaldtse commented 1 year ago

@HassanAkbar can you please help make a new release that includes this fix? Thanks.

HassanAkbar commented 1 year ago

@ronaldtse released version 0.3.31 for this fix.